Every LLM deployment makes a routing decision — whether you realize it or not. This is the story of four approaches, from simple round-robin to cache-aware intelligence, and where each one shines.
The runtime (vLLM, SGLang) is an orthogonal choice. The difference is how requests find their way to pods.
Watch three request flows side by side. Round-robin scatters requests randomly. Cache-aware routing sends them to the pod with the warmest KV cache.
Each stack builds on the one before it. The Cache-Aware stack plugs llm-d's Endpoint Picker (EPP) into the Inference Gateway via the Gateway API.
llm-d's inference scheduler also supports prefill/decode disaggregation, priority- and load-aware scoring, and wide expert parallelism — all behind the same gateway.
Illustrative TTFT curves as context length grows — same runtime (vLLM), only the routing changes. Patterned on llm-d community benchmark results.
Different workloads favor different routing. Cache-aware routing dominates when prefix overlap is high. Scores are illustrative fit ratings, not benchmarks.
An honest assessment. Cache-aware routing is not always the answer — here is when it matters and when it does not.
llm-d is an open-source CNCF Sandbox project. It runs on any Kubernetes cluster, any GPU vendor (NVIDIA, AMD, Intel), and any vLLM-supported model. The routing logic is decoupled from the infrastructure.
Performance figures in this deck are illustrative, patterned on llm-d community benchmarks (vLLM runtime).
Author: Markell Rawls