Hyperscalers distribute AI training across regions to manage power and compute limits
Major cloud providers are linking datacenters over long distances to coordinate thousands of GPUs for large-scale model training.
Key points
- Google, Microsoft, AWS, and Meta are connecting clusters across multiple geographic locations.
- Cisco estimates modern training requires tens of thousands of GPUs per cluster.
- Distributed training allows operators to bypass single-site power and space constraints.
Major technology companies are increasingly distributing artificial intelligence model training across multiple geographic datacenters rather than confining the work to single facilities. This shift addresses the physical limitations of individual sites while coordinating thousands of accelerators over hundreds of kilometers.
Google stated that its Gemini model was trained synchronously across clusters in multiple locations. Microsoft has linked AI datacenters in Wisconsin and Georgia to function as one distributed supercomputer. AWS connected wide-area clusters to support Anthropic’s Claude models, while Meta built high-capacity interconnects for training. CoreWeave and Google Cloud recently announced cross-cloud training with a private interconnect, with Azure expected to follow later this year.
Network and Infrastructure Challenges
The geographic spread of training stems from the strain on single datacenters trying to meet new compute and power demands. Cisco estimates that training models today can require clusters with tens of thousands of GPUs. Researchers at Epoch AI project that the largest frontier training runs could draw 4-16GW of power by 2030. Distributing compute allows operators to build in locations with more available power, space, and fewer planning constraints.
At the core of large language models is a neural network with billions of numerical parameters. During training, data batches pass through the model, which makes predictions. The system measures error, calculates weight changes, and updates the parameters before repeating the process. This work is distributed across thousands of GPUs and accelerators like AWS Trainium and Google TPUs.
Different GPUs process separate data batches or model parts but cannot work entirely independently. They must exchange results and synchronize at various points before the next training stage begins. The networks carry large arrays of numerical data, such as gradients and intermediate results, rather than the raw text or images used for training. Thousands of accelerators may need to exchange and combine this data simultaneously.
Who Is Affected
Cloud providers and hyperscalers face new networking difficulties when coordinating thousands of GPUs over long distances. While linking GPUs within a single datacenter is challenging, the wide-area distribution introduces latency and synchronization complexities. Operators must ensure that numerical data arrays are exchanged and combined efficiently across hundreds of kilometers to maintain training progress.
This architectural shift impacts the design of datacenter interconnects and wide-area network strategies. Companies like Microsoft and AWS are engineering solutions to treat distributed sites as a single compute resource. The need for high-capacity links between regions becomes critical as model sizes grow and power requirements exceed local grid capacities.
What should you do?
Security and technology professionals should monitor network performance metrics for signs of synchronization delays in distributed training environments. Organizations planning large-scale AI initiatives must evaluate their power and space constraints against the complexity of wide-area networking. Teams should assess how data exchange between accelerators impacts overall training efficiency and network load.
Background: Large language models
Large language models are a specific type of neural network trained on text. Generative AI is the broader umbrella term for any system that creates new content, including images and code. Not all generative AI uses large language models, but most modern text generators do.
What to do and how to stay safe: Google
- Review network architecture to ensure it can handle large-scale data exchange between distributed accelerators.
- Evaluate power and space constraints at current datacenter locations before expanding training workloads.
- Monitor synchronization points in training workflows to identify potential bottlenecks in wide-area links.
- Assess the impact of cross-cloud or multi-region training on data transfer costs and latency.
Step-by-step guide: Prompt Injection Warning Signs: How to Spot AI Manipulation
General security guidance from the Threat Times newsroom. It is not confirmed advice from the organisations named in this story.
Frequently asked questions
Why are companies moving AI training to multiple datacenters?
Single datacenters are strained by the compute and power demands of new models, with some runs potentially requiring 4-16GW of power by 2030.
What data is exchanged between GPUs during distributed training?
Networks carry large arrays of numerical data such as gradients and intermediate results, not the raw text or images used for training.
Which companies are using distributed AI training?
Google, Microsoft, AWS, Meta, and CoreWeave are connecting clusters across multiple locations or clouds to support large-scale model training.
