How llm-d's Prefix-Cache-Aware Routing Works

The Hidden Tax on Every LLM Request

Every time you send a request to an LLM, the model computes a KV (key-value) cache for every token in your prompt. For a 32,000-token document, that’s 3,700 milliseconds of GPU compute -- before the model generates a single word.

Now imagine 10 users asking questions about the same document. With standard round-robin load balancing, each request lands on a different pod. Each pod computes the full 32K-token KV cache from scratch. That’s 37 seconds of redundant GPU work -- for prompts that share 99% of their tokens.

This is the hidden tax of round-robin routing in LLM inference.

How Prefix-Cache-Aware Routing Works

llm-d’s routing algorithm solves this by tracking which KV cache segments are already computed on which pods. The Endpoint Picker Policy (EPP) maintains a real-time index of cached prefixes across all replicas. When a new request arrives, the EPP:

1. Extracts the token prefix -- the portion of the prompt that could be shared with previous requests (system prompt, document context, conversation history)
2. Queries the prefix index -- finds which pods already have matching tokens cached in GPU memory
3. Scores each candidate pod -- weighing cache match length against current load, queue depth, and available KV cache capacity
4. Routes to the best match -- the pod where the most prefill computation can be skipped

The result: on a cache hit, the prefill phase is effectively free. The model skips straight to generating the response.

Where It Matters Most

Multi-turn conversations

In a 5-turn conversation, turn 5 reuses 100% of turns 1–4. With cache-aware routing, only the new user message needs prefill. Every previous turn is a cache hit. The longer the conversation, the bigger the savings.

Agentic workflows

An AI agent with 10 parallel tool calls shares the same system prompt and conversation context across all 10 requests. With round-robin routing: 10 cold prefills. With llm-d: 1 cold prefill + 9 cache hits.

Long-context document Q&A

Upload a 32K-token contract and ask 20 questions about it. Each question shares the full document prefix. With cache-aware routing, questions 2–20 skip 3,700ms of prefill -- each responding in under 80ms.

The Numbers

Benchmarked on 3x NVIDIA H200 GPUs running Llama 3.1 70B (FP8) on Red Hat OpenShift. The first request to each unique prefix is a cold miss -- every subsequent request with a matching prefix is a hit.

What This Means for Your GPU Bill

Skipping prefill doesn’t just improve latency -- it frees GPU cycles. Every cache hit is compute you didn’t burn. At scale, this translates directly to:

• Fewer GPUs needed for the same throughput
• Lower cost per token as redundant compute is eliminated
• Higher effective utilization of existing hardware

How to Try It

llm-d is open source and deploys on any Kubernetes cluster. The interactive routing visualization shows exactly how the EPP makes routing decisions -- watch requests flow to cached pods in real time.

• llm-d on GitHub: github.com/llm-d/llm-d
• What is llm-d? redhat.com/en/blog/what-llm-d-and-why-do-we-need-it
• Community: #llm-d on CNCF Slack