Benchmarking cache aware routing on NVIDIA H200 GPUs

If you're serving large language models on Kubernetes today, your inference stack probably has a problem you haven't noticed yet. Every follow-up message in a multi-turn conversation, every tool call in an agentic chain, every iterative question against a long document. All of them are recomputing context that was already computed seconds ago on a different pod.

This is the prefill problem. Before an LLM generates a single token, it has to process every prior token in the conversation. This prefill phase scales linearly with context length and dominates time to first token (TTFT). Standard Kubernetes round robin load balancing makes it worse by scattering sequential requests across pods and forcing each one to re-derive the KV cache from scratch.

llm-d solves this with KV cache aware routing. Its Endpoint Picker sends each request to the pod that already has the relevant KV cache in GPU memory. Instead of recomputing, it reuses.

We benchmarked this on real hardware to find out how much it actually matters. Turns out? A lot more than we expected.

What we tested

We designed a benchmark suite to isolate the impact of cache aware routing under realistic inference conditions. Every measurement was taken inside the pod so the numbers reflect what the GPU is actually doing, not what the network adds.

Model

Meta Llama 3.1 70B Instruct with FP8-dynamic quantization, published by Red Hat AI on Hugging Face

Hardware

3x NVIDIA H200 141GB HBM3e GPUs on a single Red Hat OpenShift node

Inference engine

vLLM with automatic prefix caching enabled

Measurement

In-pod via direct HTTPS with streaming SSE parsing. time.perf_counter() for TTFT. Zero network overhead

Routing comparison

Round-robin: strict pod rotation simulating standard Kubernetes Service behavior. Cache-aware: all session requests routed to one pod, simulating llm-d's EPP

Test suite

8 tests covering context scaling (500 to 32K tokens), concurrent load, agentic chains, multi-turn conversations, and cache warmup dynamics

The scaling advantage

Cache aware routing gets faster relative to round robin as context length increases. Cold start TTFT scales linearly at roughly 0.095ms per token. Cache hit TTFT? Barely moves.

That growing gap between those two columns? That's the entire value proposition of cache aware routing, and it just keeps growing with every token of context.

At 32,000 tokens (common for RAG with long documents), a cold start request waits 3,742 milliseconds before the model generates a single token. With cache aware routing, the same request produces its first token in 79 milliseconds.

As models move to 128K context windows, this advantage only grows. The curve showed no sign of plateauing at 47.5x.

Why this pattern matters

Look at what happens in that middle range. At 8,000 tokens, you're already seeing a 17x speedup. This would be things like a typical RAG retrieval with a few documents, a customer support agent pulling up account history and recent tickets, or a code completion model with the current file and a few related imports.

By 16,000 tokens, you're at 29.6x. This is where you start seeing multi-document analysis like research assistants comparing papers, legal document review, or technical documentation generation that pulls from multiple source files.

At 32,000 tokens, you hit 47.5x. This is where the heavy workloads live. Entire codebases for context aware refactoring, long form document Q&A, multi-turn debugging sessions where the entire conversation history matters.

The pattern holds. Double the context, roughly double the speedup advantage. Your users either get instant responses or they watch spinners.

Under production load

Single-request benchmarks are useful, but production inference servers handle many sessions at once. We ran 10 concurrent sessions, each executing 5 conversational turns simultaneously.

Round robin delivered a mean TTFT of 220ms with a P95 of 288ms. Cache aware routing delivered a mean of 55ms and P95 of 73ms. That's a 75% reduction.

The advantage was larger under concurrent load, not smaller. Cache misses waste GPU compute on redundant prefill, and that wasted work competes with other in-flight requests for GPU cycles. Cache hits free the GPU to focus on decoding, which benefits everyone in the queue.

Cache aware routing stayed flat at 52 to 61ms across all 5 turns. Round robin oscillated between 179 and 289ms depending on whether a given turn happened to land on a pod that had already seen this session. Variance matters just as much as absolute delay.

What this means for cost

Every millisecond of prefill is GPU time you're paying for. Round robin at 32K context burns 3,742ms of GPU compute per request. Cache aware routing uses 79ms. That's 47x less GPU compute per request with the same hardware, same model, same output quality.

At 1,000 requests per hour with 32K context, round robin burns over an hour of GPU time per hour of wall clock time just on prefill. Cache aware routing cuts that to about 1.3 minutes, which means the rest of your GPUs can be doing actual generation instead of recomputing the same prefix 47 times.

Agentic workflows compound the benefit

We simulated a 12 step agentic chain processing a 7,000 token document. Context grew from 7,000 tokens at step 1 to 10,300 tokens by step 12 as tool calls and reasoning steps accumulated. RAG retrieval followed by iterative analysis, summarization, and action planning. This pattern shows up constantly in production.

Under round robin, the first 3 steps averaged roughly 700ms each. That's the cold start penalty, with each step landing on a pod seeing the document for the first time. Steps 4 through 12 dropped to about 145ms as the prefix cached across all pods, but each pod still re-prefilled the growing conversation context beyond the shared prefix.

Cache aware routing stayed at 44 to 87ms throughout the entire chain. Every step landed on the same pod, so each step only prefilled the roughly 250 new tokens added since the last step. We saw a 40% TTFT reduction even in late stages where round robin had already warmed its prefix caches.

In an agentic workflow, every tool call carries the entire prior conversation as context. Without cache aware routing, each step re-prefills everything the previous step already computed on a different pod.

What this means at fleet scale

Our benchmark used 3 pods. That's enough to demonstrate the pattern, but in production fleets with tens or hundreds of pods, the cache aware routing advantage becomes even more pronounced.

With round robin routing, each pod maintains its own independent cache. As fleet size grows, cache dilution occurs. Each pod caches a smaller fraction of the total working set. This means cache hit rates decrease as you scale, even though individual cold start latency remains around 3,742ms at 32K context.

Cache-aware routing maintains consistent performance regardless of fleet size. By intelligently routing requests to pods with matching cached prefixes, the effective cache hit rate stays high even as the fleet scales to hundreds of replicas. Each session sees the same ~79ms warm performance whether you're running 3 pods or 300.

The key insight: round robin's cache hit rate degrades with fleet size due to dilution across more pods. Cache-aware routing maintains high hit rates by directing each session to the pod that already has its context cached.

Where it doesn't help (and that's okay)

When 20 users queried the same shared document simultaneously, sticky routing was 15.7% slower than round robin with 3 pods. All 3 pods quickly cached the shared prefix through natural request distribution, so sticky routing just concentrated load on one pod unnecessarily.

This is actually the correct behavior. Cache aware routing's value is in per session context. Conversations, agent chains, user specific RAG retrieval. For globally shared prefixes that all pods cache quickly, the EPP should distribute load evenly. And at larger fleet sizes (50+ pods), even shared prefixes benefit from cache aware routing because fleet wide warmup takes proportionally longer.

In production, expect even more

Several factors would make the advantage more dramatic in production than what we measured on our 3 pod testbed:

• Fleet size: With 50 to 100+ pods, cache dilution under round robin becomes severe. Cache-aware routing maintains high hit rates regardless of fleet size
• Context windows: 32K to 128K tokens extend the speedup curve further.our tests showed no sign of plateauing at 47.5x
• Real EPP intelligence: The actual llm-d Endpoint Picker uses prefix hashing to find the pod with the best cache match across all sessions, not just sticky routing to one pod
• Mixed workloads: Production traffic combines agentic chains, multi-turn conversations, and RAG simultaneously, compounding the benefits we measured in isolation

Some factors temper the results. The EPP adds approximately 1 to 2ms of routing latency. Short contexts under 500 tokens see a smaller absolute savings (2.7x rather than 47x). And single-shot completions with no context reuse see minimal benefit.

What's next

llm-d is an open source project under the Cloud Native Computing Foundation (CNCF) sandbox. The full source, documentation, and deployment guides are available on GitHub.

All links referenced in this post