Meta’s Prometheus Supercluster Reaches Phase One Completion in Ohio
Meta confirms the first phase of its gigawatt-scale AI supercluster is finished, having trained the new Muse family of foundational models with high stability.
Key points
- The Prometheus supercluster in New Albany, Ohio, has completed its first phase of construction.
- Infrastructure achieved over 85% goodput for pre-training runs of the Muse models.
- Meta plans to scale the facility to approximately one million GPUs across multiple buildings.
Meta has completed the initial phase of Prometheus, its gigawatt-scale supercluster located in New Albany, Ohio, as detailed in a recent post on Engineering at Meta. This infrastructure milestone enabled the training of the company’s new Muse family of foundational models, including the Muse Wonder variants, which are designed for agentic tasks.
The completed phase supports hundreds of thousands of GPUs distributed across interconnected buildings, functioning as a single machine. Meta is currently working toward scaling the supercluster to comprise roughly one million GPUs. The facility utilizes closed-loop liquid cooling and miles of fiber interconnects to maintain efficiency and performance for large training jobs.
Technical Architecture and Reliability
According to Engineering at Meta, the supercluster deploys two NVIDIA GPU platforms, with a third based on NVIDIA’s Vera Rubin NVL72 expected to arrive soon. The first platform, Catalina, leverages the NVIDIA GB200 NVL72 in an Ariel variant configuration. This setup pairs each NVIDIA Blackwell GPU with an NVIDIA Grace CPU and two NVIDIA ConnectX-7 400G NICs, delivering 800G of I/O per GPU.
The Ariel variant uses a two-rack configuration where 36 GPUs per rack are interconnected via NVLink to form a unified 72-GPU domain. Meta chose this configuration because it scales up FLOPS, HBM bandwidth, and scale-up domain size relative to the NVIDIA H100, while providing improved scale-out bandwidth per GPU compared to standard configurations. The infrastructure builds upon the Grand Teton H100 generation, offering improvements in raw compute and memory capacity.
Operational Performance Metrics
Large training jobs require optimal system stability to sustain performance, as a single host issue can slow or stop the entire job, lowering goodput. Meta reported achieving more than 85% goodput for pre-training runs and more than 90% goodput for reinforcement learning runs of Muse Wonder. These metrics reflect the company’s focus on improving system stability and reliability during the training of frontier models.
To measure interruption frequency, Meta tracked the wall-clock mean time between interruptions. During initial training runs, the fleet achieved an MTBI of approximately 47k GPU hours. Since those runs, Meta implemented firmware fixes across the fleet, increasing the MTBI to more than 54k GPU hours. The company states it is constantly improving these stability metrics as it continues to scale the infrastructure.
What should you do?
Security and technology professionals should monitor infrastructure updates from major cloud providers for changes in hardware configurations that may affect compatibility or security postures. Organizations relying on AI services should assess how underlying hardware shifts, such as the move to NVL72 architectures, impact their own deployment strategies and latency requirements. Teams should review their own GPU cluster management practices to ensure they are implementing similar stability and monitoring measures to maintain high goodput during large training jobs.
What to do and how to stay safe: Meta
- Review your organization’s GPU cluster monitoring tools to ensure they can detect single-host issues that may degrade overall training job performance.
- Assess the compatibility of your current AI workloads with emerging hardware architectures, such as NVIDIA’s NVL72 rack configurations, to plan for future infrastructure upgrades.
- Implement firmware update procedures that prioritize stability and reliability, similar to the fixes Meta applied to increase mean time between interruptions.
- Monitor vendor announcements regarding new GPU platforms and networking standards to anticipate changes in bandwidth and compute capabilities that may affect your strategic planning.
Step-by-step guide: Mobile Device Security: How to Protect Data Beyond Passwords
General security guidance from the Threat Times newsroom. It is not confirmed advice from the organisations named in this story.
Frequently asked questions
What is the Prometheus supercluster used for?
The Prometheus supercluster is used to train foundational AI models, including Meta’s Muse family and Muse Wonder models, for next-generation AI experiences.
How many GPUs will Prometheus have at full scale?
At full scale, the Prometheus supercluster will comprise roughly one million distributed GPUs, all functioning in lockstep across multiple buildings.
What GPU platforms does Prometheus currently deploy?
Prometheus currently deploys two NVIDIA GPU platforms, with a third based on NVIDIA’s Vera Rubin NVL72 arriving soon, including the Catalina platform using GB200 NVL72.
