Microsoft unveils Maia 300 AI accelerator to challenge NVIDIA at gigawatt scale
The new chip uses a dataflow architecture to process 40% more tokens per megawatt than leading GPU-based systems.
Key points
- Microsoft’s Maia 300 uses four chiplets on TSMC’s 2-nanometer node with 288GB of HBM4 memory.
- The accelerator employs a dataflow architecture to minimize data movement and improve energy efficiency.
- Microsoft plans to deploy the chips in liquid-cooled racks containing 96 accelerators for massive scale.
Microsoft officially unveiled the Azure Maia 300, its next-generation custom silicon designed for artificial intelligence inference workloads, according to reports from Neowin and SiliconANGLE. The company stated that this internally developed accelerator will serve as its primary source of inference capacity over time, aiming to power both internal operations and partner systems at multi-gigawatt scales.
The Maia 300 is optimized specifically for inference tasks rather than model training. Microsoft estimates that the chip can process 40 percent more tokens per megawatt than leading GPU-based systems, a comparison that likely references products from NVIDIA Corporation. This efficiency gain is central to Microsoft’s strategy for scaling AI infrastructure without exponentially increasing power consumption or cooling requirements.
Technical Architecture and Design
The Maia 300 features a multichiplet design comprising four computing chiplets manufactured using Taiwan Semiconductor Manufacturing Co.’s latest two-nanometer process node. These computing units are paired with eight HBM4 memory modules, providing a combined capacity of 288 gigabytes. This memory capacity matches that of NVIDIA’s flagship Rubin accelerator, although Microsoft noted that the Maia 300 offers approximately 20 percent less memory bandwidth than the competing NVIDIA hardware.
To address the performance limitations associated with lower memory bandwidth, Microsoft implemented a specialized dataflow architecture. Traditional AI chips often slow down because they repeatedly shift data between computing circuits and memory. The Maia 300 minimizes this movement by keeping information in an internal SRAM cache whenever possible, rather than sending it to the external HBM stacks.
When access to HBM is unavoidable, the chiplets send data to the nearest memory stack to reduce transmission latency. Additionally, the system compresses data into lightweight formats such as MXFP4. Microsoft executives Rani Borkar and Saurabh Dighe explained in a blog post that these narrower representations reduce memory footprint and data movement, enabling the system to support larger models and longer contexts within the same power envelope.
Deployment and Infrastructure
Beyond the computing and memory components, the Maia 300 includes a device control chiplet that manages power flow, collects telemetry, and performs other supporting tasks. Two additional auxiliary chiplets handle the movement of data in and out of the accelerator, ensuring efficient communication with the broader system network.
Microsoft plans to deploy the Maia 300 in trays that house four accelerators each. These trays will be installed in liquid-cooled racks, with each rack containing a total of 96 accelerators. This dense configuration supports the company’s goal of operating AI workloads at gigawatt scale, leveraging liquid cooling to manage the significant heat output generated by high-performance inference tasks.
What should you do?
Organizations relying on cloud-based AI inference should monitor Microsoft’s Azure updates for availability of Maia 300-powered instances. Security and infrastructure teams should evaluate the energy efficiency metrics of their current GPU-based inference workloads to determine if migrating to new custom silicon could reduce operational costs. IT planners should also review their cooling infrastructure requirements, as the shift to dense, liquid-cooled racks may impact data center design standards.
What to do and how to stay safe: Microsoft
- Review current AI inference workload costs and energy consumption to benchmark against new efficiency claims.
- Monitor Azure documentation for specific instance types powered by the Maia 300 accelerator.
- Assess data center cooling capabilities to ensure compatibility with high-density, liquid-cooled rack deployments.
- Evaluate the impact of reduced memory bandwidth on specific model types before migrating workloads.
General security guidance from the Threat Times newsroom. It is not confirmed advice from the organisations named in this story.
Frequently asked questions
How does Maia 300 compare to NVIDIA’s Rubin accelerator?
Maia 300 has the same 288GB memory capacity as Rubin but offers about 20% less memory bandwidth.
What manufacturing process is used for the Maia 300 chiplets?
The computing chiplets are made using TSMC’s latest two-nanometer node.
How many accelerators are in a single Maia 300 rack?
Each liquid-cooled rack contains a total of 96 accelerators.
