When two requests share the same beginning, the KV cache computation is identical. llm-d's EPP knows what each pod has cached -- and routes to the pod that already has the answer half-computed.
Every LLM prompt computes a massive matrix of attention states -- the KV cache. Standard load balancers scatter requests randomly, forcing every pod to redo this computation from scratch.
Request 1 goes to Pod A, which computes the full KV cache. Request 2 -- with the exact same system prompt -- goes to Pod B, which computes it all over again. The cache sitting in Pod A's GPU memory is wasted. Multiply this across every request and you are burning GPU cycles recomputing attention states that already exist.
The EPP (Endpoint Picker Policy) knows what each pod has cached. When Request 2 arrives with the same prefix, the EPP routes it to Pod A -- the pod that already has the cache. Pod A skips the entire prefill phase and goes straight to generating tokens. Zero redundant compute. The longer the shared prefix, the bigger the savings.
Cache-aware routing in action -- auto-playing now
Four steps from request to routed response. The EPP makes this decision in microseconds, before the request ever reaches a vLLM pod.
A chat completion request hits the Gateway API. The EPP intercepts it and extracts the token prefix -- the system message plus conversation history.
The EPP maintains a near-real-time map of what each pod has cached. It checks every replica's cached prefix blocks against the incoming request.
Each pod gets a score based on the longest prefix match -- how many contiguous token blocks from the start of the request are already in that pod's KV cache.
The request goes to the highest-scoring pod. On a full cache hit, the pod skips prefill entirely and goes straight to generating new tokens.
Prefix-cache-aware routing delivers the biggest gains in workloads with high prefix overlap. These three patterns are where the math works hardest in your favor.
Each turn builds on the previous. Turn 3's prompt contains all of turns 1 and 2 as prefix. Routing to the same pod means only the new message needs prefilling.
Agent frameworks send parallel requests sharing the same system prompt. First call computes the cache; every subsequent call to the same pod skips system prompt prefill.
Multiple questions about the same document share identical document tokens. A 32K-token document cached means every follow-up skips 32K tokens of prefill.
Five requests with the same system prompt hit a 3-pod cluster. Watch the difference.
Requests scatter across pods. Each pod computes the system prompt KV cache independently.
EPP routes all requests with the same prefix to the pod that already has the cache.
From conference halls to Slack channels -- the questions people ask most.
No. The KV cache lives in each pod's GPU memory. Pod A cannot read Pod B's GPU. This is exactly why routing matters -- send the request to the pod that already has it.
There is no timer. vLLM uses LRU eviction -- blocks persist until GPU memory fills up and pushes out the least-recently-used blocks. Under consistent traffic, caches last indefinitely.
The EPP scoring adds microseconds -- a hash comparison across a small set of pods. The savings from skipping prefill (hundreds of ms to seconds) dwarf the routing cost.
The EPP falls back to load-based routing. That first request computes the cache, and all subsequent requests with the same prefix benefit from it.
Any model served by vLLM with automatic prefix caching enabled (the default). Llama, Mistral, Qwen -- any transformer. The cache operates at the KV block level.
No. Cache-aware routing is a routing-layer optimization. Swap in llm-d as your inference gateway and cache reuse happens automatically -- no SDK changes needed.
By routing with awareness of what each pod already knows, llm-d transforms redundant GPU compute into instant cache hits. Faster time-to-first-token. Lower cost per request. Infrastructure that gets smarter the more consistent your workloads become.