Microsoft plans to scale its Azure Maia accelerator family to multi-gigawatt deployments starting in 2027. The company says Maia will become the primary inference system for its products and services. Its new Maia 300 design puts memory and data movement at the center of that plan, while its headline efficiency claim has drawn questions.
Maia 200 is already serving Microsoft Copilot and the MAI team in the United States, Europe, and Australia. Microsoft plans to add GitHub Copilot and Microsoft Foundry by the end of the year. It says Maia 200 delivers up to 1.7 times more tokens per watt for built-in AI in Teams and Copilot than leading GPU infrastructure. Maia 300 production deployments begin in 2027.
Maia 300 puts memory close to inference work
Maia 300 targets long-context, memory-intensive, and decode-heavy inference. Microsoft did not describe a training role. Each accelerator has eight HBM4 stacks with 288 GB of memory and 18 TB/s of bandwidth within a 950-watt TDP. A 167 kW rack holds 96 accelerators in 24 four-accelerator trays. Together, they provide 27.6 TB of HBM and 1.7 PB/s of aggregate bandwidth.
Compute chiplets use TSMC N2P, while memory-controller and I/O functions sit on separate dies. Microsoft says the design raises compute FLOPs by 36% and contributes to 2.5 times the memory bandwidth and roughly twice the HBM bandwidth per watt of Maia 200. A NUMA memory layout lets software place data near the chiplet that uses it.
Read nextVolvo plans Nvidia-powered hands-free driving by 2030Maia’s AI network controller runs a custom transport over Ethernet. Two accelerators link directly within a tray. Switches connect all 96 in a rack, and optical switches connect up to 128 racks in a superpod of 12,288 accelerators. Microsoft describes direct pairs, one-hop connections within a pod, and three-hop connections across pods. It expects software to keep tensor-parallel traffic in the lowest-latency domain.
Accelerators can communicate with peers without a host or separate scale-out network card. Job placement determines how bandwidth is split within and across racks. When only some traffic crosses racks, an accelerator can send up to 750 GB/s to any peer. After a prefill worker finishes, KV state moves over the same fabric to a selected decode worker. The fabric also lets a spare tray in one rack cover for a failed tray in another.
Microsoft’s efficiency comparison draws questions
Microsoft expects Maia 300 to deliver 40% more tokens per megawatt than leading GPU-based systems. It also projects up to seven times the rack-scale token throughput and more than 2.5 times the performance per watt of Maia 200. The company does not identify the GPU systems in its comparison or define what it means by a “useful” token. Scott Guthrie and the engineering post describe the 40% gain as an expectation; Satya Nadella’s post states it as a delivered result.
Commenters questioned the metric and the workload behind it. Rick Bullotta argued that tokens are not a uniform measure of compute and that the metric could reward verbosity. Michel Floyd asked, “Shouldn’t it either be tokens per second per watt or tokens per KWh?” Vinay Bansal, founder of competing inference accelerator company UBIQS, asked what context length Microsoft used. He pointed to memory demands in sustained, multi-step agent workloads.
Access is another open question. The workloads Microsoft names, including Dragon Copilot, are its own. Neither announcement says customers can provision Maia capacity directly. Pexon Consulting chief executive Phillip Pham asked whether the gains would lower the price of provisioned throughput reservations or stay within Microsoft’s services. Microsoft has published an engineering blog post by Rani Borkar and Saurabh Dighe, but no separate technical paper. It says it will share more at Microsoft Ignite 2026 in San Francisco next month.
