Google has released llm-d 1.0, an open-source stack for teams serving their own AI models across multiple servers. Built for Kubernetes, it routes requests to servers that already hold useful context and controls how interactive and background work share capacity. The release supports Google Cloud TPUs and GPUs.
The stack works with the vLLM and SGLang model-serving engines. Its router tracks which servers hold cached parts of earlier prompts, then considers load and predicted latency when assigning a request. Google says this can reduce repeated processing during multi-step agent tasks.
Traffic controls give interactive requests priority
Version 1.0 adds priority and fairness controls for teams running different workloads on the same fleet. An asynchronous processor can pull background requests from Google Cloud Pub/Sub or Redis queues and dispatch them when capacity is available. Session-aware routing and scheduling also help agents reuse context across turns.
A new DisaggregatedSet component manages separate pools for the stages that process prompts and generate responses. Google says it coordinates rolling updates across those pools without dropping context or disrupting active sessions.
Google reports results from its deployments
Google says prefix-aware routing and host-memory cache offload cut median time to first token by more than 85% and raised throughput 2.4 times on an agentic coding workload using Ironwood TPU, compared with a plain Kubernetes Service as of October 2026. The company also says its Gemini platform doubled its prefix cache hit rate from 35% to 70% while serving production traffic through GKE Inference Gateway.
Google co-founded llm-d with Red Hat, IBM and others. It says more than 500 contributors across 25 organizations have worked on the Cloud Native Computing Foundation Sandbox project. Teams can adopt individual serving components instead of deploying the entire stack.
llm-d 1.0 is available now. Google says teams can deploy it on Google Kubernetes Engine using verified Helm charts, or start by adding prefix-aware routing to an existing vLLM deployment.
