

Google has published a November 19 walkthrough of Streaming DiLoCo in MaxText, its open-source JAX training framework. The implementation targets a specific scaling problem: accelerator groups can compute efficiently within a site yet spend much of their time waiting for slower connections between sites.
The primary article explains how local optimization and staggered synchronization reduce that traffic bottleneck. It is an implementation and evaluation update for distributed training, not a claim that a desktop can train a frontier model or that ordinary home-network bandwidth is sufficient.
Fewer exchanges, spread across more steps
Conventional data-parallel training synchronizes gradients repeatedly across participating groups. DiLoCo lets groups perform several local optimization steps before exchanging model changes. Its streaming variant divides those changes into fragments and schedules the exchanges over time, reducing the large periodic burst produced by synchronizing the whole model at once.
Communication inside a learner group still occurs. The technique changes the outer synchronization pattern; it does not eliminate the need for fast accelerator interconnects, memory or compute within each group. The project's theory documentation and implementation are available for readers to inspect.
The benchmark conditions explain the headline
Google evaluates Qwen3-32B across two TPU v5p-32 slices, using network shaping to constrain the inter-slice connection. Under its 10-Gbit/s condition, the article reports approximately 5.76 seconds per step for Streaming DiLoCo against 55.11 for the data-parallel comparison. On unthrottled links, the methods have similar step times. We have not reproduced these measurements.
Our analysis: the important result is resilience to a bottleneck, not a free speedup on every cluster. When communication is already hidden behind computation, reducing it may change little. When it dominates a step, a better synchronization schedule can change whether additional compute is useful at all.
Convergence also needs its own evidence. A faster step is not enough if more steps are required or if the resulting model behaves worse. Google's reported loss comparisons are evidence for its evaluated settings; they do not settle every architecture, optimizer or workload.
MaxText is published under Apache-2.0, while training data and checkpoints retain their own terms. Our MaxText repository profile links the upstream code, and the Qwen3-32B profile identifies the evaluated model. These links are for inspection, not a tested cloud deployment recipe or a promise about a reader's hardware.
EXPLORE THE SOURCE
Projects in this story
GitHub repositories
- AI-Hypercomputer/maxtext ↗View project details
Original primary-source review and clearly labeled editorial analysis. No local model evaluation, Mac automation, cloud training, software installation or independent benchmark was performed.



