Meta keeps thousands of idle GPUs at Prometheus to offset hardware failures
More than 11 percent of accelerators at Meta’s Ohio supercluster sit idle or are dead, the company revealed in a new engineering blog post this week.
Key points
- Meta reports over 11 percent of GPUs at its Prometheus supercluster are idle or failed.
- The company reduced training interruptions by 99 percent in the past year.
- The cluster uses Nvidia Grace Blackwell systems and aims for one million accelerators.
Meta disclosed this week that more than 11 percent of the graphics processing units at its Prometheus supercluster in New Albany, Ohio, are either non-functional or sitting idle as reserves. According to a blog post published by the social media giant, this massive pool of redundant hardware is a necessary measure to maintain stable AI model training runs across its expanding infrastructure.
The idle units serve as backups to step in when active chips fail, a common occurrence in large-scale compute environments. Meta revealed this strategy while discussing how it keeps its training pipelines on track for foundational models. The company noted that maintaining stability is critical for long-duration training jobs that span multiple datacenters and require consistent performance from thousands of interconnected accelerators.
Scale and Stability Challenges
Prometheus represents one of Meta’s largest compute deployments to date, powered by Nvidia’s Grace Blackwell rack systems. The supercluster is designed to eventually scale to a gigawatt of capacity, spanning approximately one million accelerators. As Threat Times reported on October 20, 2026, Meta previously confirmed the first phase of this gigawatt-scale supercluster in Ohio was complete, having trained the new Muse family of foundational models with high stability.
However, scaling to such extreme levels introduces significant reliability challenges. In a standard eight-GPU node, the probability of a single GPU failing is relatively low. But in a cluster containing 100,000 GPUs or more, hardware failures become unavoidable. Meta is building a system ten times larger than that threshold, making distributed training runs across multiple facilities a major engineering hurdle.
Reducing Interruptions
When Meta began the first phase of its Prometheus deployment, the company experienced roughly 1,000 interruptions per day for every 8,000 GPUs. These interruptions do not necessarily indicate permanent GPU failure; rather, they signify that a component crashed, failed, or timed out, forcing the training run to stop and roll back to the last known good checkpoint.
In the year since launching the initial phase, Meta claims it has cut down the number of these daily interruptions by 99 percent. The current rate stands at about 11 interruptions per day for every 8,000 GPUs. This improvement was achieved partly by keeping a significant portion of Prometheus’ GPUs in reserve to handle unexpected hardware drops without halting the entire training job.
What should you do?
Security and technology professionals should monitor how hyperscalers manage hardware redundancy in AI clusters, as these strategies impact overall system availability and attack surfaces. Large-scale deployments require robust monitoring for component failures that could be masked by redundant systems. Understanding the baseline failure rates of high-density GPU clusters helps in planning for similar infrastructure.
What to do and how to stay safe: Meta
- Monitor system logs for frequent checkpoint rollbacks, which may indicate underlying hardware instability in large GPU clusters.
- Evaluate the ratio of idle to active accelerators in your infrastructure to ensure redundancy does not create unnecessary maintenance overhead.
- Implement automated failover mechanisms that can seamlessly replace failed nodes without requiring manual intervention during training runs.
- Review vendor documentation for expected failure rates in high-density rack systems to set realistic expectations for system uptime.
Step-by-step guide: Mobile Device Security: How to Protect Data Beyond Passwords
General security guidance from the Threat Times newsroom. It is not confirmed advice from the organisations named in this story.
Frequently asked questions
How many GPUs are idle at Meta's Prometheus supercluster?
More than 11 percent of the GPUs at Meta’s Prometheus supercluster are either dead or sitting idle waiting to replace failed units.
What caused the reduction in training interruptions at Prometheus?
Meta reduced interruptions by 99 percent by keeping a large pool of reserve GPUs to handle failures without stopping training runs.
What hardware powers the Prometheus supercluster?
The cluster is powered by Nvidia’s Grace Blackwell systems, specifically GB200 and GB300 units, with plans to add Vera Rubin systems later.
