Google Cloud, Red Hat and IBM Launch llm-d 1.0 for Distributed LLM Inference

Google Cloud, Red Hat and IBM released llm-d 1.0, marking the first stable version of their open-source distributed inference stack. The team unveiled the software at the PyTorch Conference North America, targeting self-hosted agent and reinforcement learning workloads where traditional round-robin load balancing often fails to maintain performance.

The Kubernetes-native layer coordinates vLLM or SGLang model servers into a single pool, enabling multi-tenant flow control and agent-aware routing. According to MarkTechPost, the release focuses on reliable production operations, which IBM refers to as day 2 operations. It supports NVIDIA, AMD and Intel GPUs alongside Google TPUs, offering a stable foundation for complex inference tasks.

Technical Capabilities and Performance

llm-d 1.0 introduces cache-aware routing that tracks which replicas hold cached prefixes, weighing load and predicted latency before selecting a target. This approach allows agents to call a single endpoint while ensuring each step lands on a replica where its context already resides. When accelerator memory fills, the system moves key-value cache data to host memory, then to SSD or Managed Lustre storage.

Vendor-reported benchmarks from a 17,000-node multi-region GKE deployment show significant performance gains. On an Ironwood TPU coding workload, median time to first token dropped by more than 85%, with throughput increasing by 2.4 times. However, reinforcement learning time-slicing only lifted accelerator duty cycles above 70%, leaving roughly 30% idle time, according to the reporting.

Project Governance and History

The llm-d project was founded in 2025 by Red Hat, IBM, Google Cloud, CoreWeave and NVIDIA. Red Hat, IBM and Google Cloud serve as the lead maintainers for the initiative. The project accepted into the CNCF Sandbox program on March 12, 2026, and operates under the Apache 2.0 license. It now counts more than 500 contributors from over 25 organizations, reflecting broad industry participation in its development.

As Threat Times reported on October 21, 2026, Google Cloud has faced widespread disruptions in Europe-West 2 affecting banking and fintech services. While that outage impacted general cloud availability, the new llm-d release aims to provide more resilient inference infrastructure through features like zero-downtime upgrades and router high availability with hot standby mechanisms.

What should you do?

Organizations running large language model inference workloads should evaluate whether their current load balancing strategies are sufficient for agent-based applications. The new llm-d 1.0 release offers specific optimizations for these scenarios, particularly regarding context retention and latency reduction. Users should review the Apache 2.0 licensed code and benchmark results to determine if the cache-aware routing benefits apply to their specific hardware configurations.

Background: Large language models

Large language models are a specific type of neural network trained on text. Generative AI is the broader umbrella term for any system that creates new content, including images and code. Not all generative AI uses large language models, but most modern text generators do.

Read the full guide: Large Language Models vs Generative AI: The Distinction and Security Implications

What to do and how to stay safe: Google Cloud

• Review your current Kubernetes inference stack to identify if round-robin load balancing is causing latency spikes for agent-based workloads.
• Evaluate the hardware requirements for llm-d, ensuring compatibility with your existing NVIDIA, AMD, Intel GPUs or Google TPUs.
• Monitor the CNCF Sandbox project repository for updates on v1 API stability and zero-downtime upgrade procedures.
• Assess your key-value cache offloading needs to determine if moving data to host memory or SSD aligns with your storage infrastructure.

Step-by-step guide: Prompt Injection Warning Signs: How to Spot AI Manipulation

General security guidance from the Threat Times newsroom. It is not confirmed advice from the organisations named in this story.

Frequently asked questions

What is llm-d 1.0?

llm-d 1.0 is the first stable version of an open-source, Kubernetes-native stack for distributed large language model inference, released by Google Cloud, Red Hat and IBM.

Which hardware does llm-d support?

The stack runs on Kubernetes with NVIDIA, AMD or Intel GPUs, as well as Google TPUs, utilizing vLLM or SGLang model servers.

Is llm-d free to use?

Yes, the project is licensed under Apache 2.0 and is a CNCF Sandbox project with over 500 contributors from more than 25 organizations.