KV-Cache-Aware Inference Routing
estimated annual savings with llm-d
Estimate how many GPUs you could free up by switching from round-robin load balancing to llm-d's KV-cache-aware routing. All figures are directional estimates built on the editable assumptions below — tune them to your workload.
Adjust the parameters below to match your deployment. Results update instantly.
See exactly which GPUs you can decommission with cache-aware routing.
Click a card to auto-fill the calculator with a realistic deployment profile.
Full transparency on how we calculate these numbers.
GPU rates are illustrative assumptions in the ballpark of 2025 cloud on-demand list pricing, not quotes: B200 at $6.00/hr ($52,560/yr), H200 at $4.50/hr ($39,420/yr), H100 at $3.50/hr ($30,660/yr), MI300X at $4.00/hr ($35,040/yr), A100 at $2.50/hr ($21,900/yr), L40S at $1.67/hr ($14,600/yr), A10G at $1.25/hr ($10,950/yr). Annual cost = hourly rate × 8,760 hours. Substitute your negotiated, reserved, or spot rates — absolute dollars change, but the relative savings ratio stays the same.
GPUs per replica scale with model size: 7B, 8B, and 13B models fit on 1 GPU, 34B needs 2 GPUs (tensor parallel), 70B needs 4 GPUs, and 405B needs 8 GPUs. Illustrative throughput per replica scales inversely with model size: 7B ~18 req/s, 8B ~16, 13B ~10, 34B ~5, 70B ~3, and 405B ~1 req/s, calibrated at ~1,024 tokens per request (load scales linearly with token count). Replicas include 30% headroom for bursts. Savings are computed at replica granularity — a tensor-parallel replica is decommissioned whole, never one GPU at a time — so small fleets can show zero savings.
The savings factor models how much effective compute prefix reuse can eliminate, and it varies by workload. Multi-turn Chat: 40% (shared conversation history). Agentic: 50% (repeated tool schemas and context). RAG: 30% (shared system prompts, varied documents). Code Completion: 35% (shared file context). Batch: 15% (minimal prefix overlap). These are illustrative planning defaults, not benchmarks — actual reuse depends entirely on your traffic's prefix overlap, so measure your own hit rates before committing capacity.
Time-to-first-token improvements come from skipping prefill computation for cached prefixes. The illustrative multipliers used here — Multi-turn Chat: 4x, Agentic: 5x, RAG: 3.5x, Code: 3x, Batch: 2x — apply to the prefill phase only when a request hits a warm cache; total end-to-end latency improvement depends on generation length and hit rate.
CO₂ estimates use approximately 2.8 metric tons per GPU per year, based on average US data center power usage and carbon intensity. Actual figures vary significantly by region, PUE, and energy mix. This is a directional estimate, not a precise measurement.
llm-d is an open source, Kubernetes-native distributed inference serving stack built on vLLM. Through the Inference Gateway (Gateway API), its endpoint picker applies KV-cache-aware, prefix-aware, and load-aware routing to send requests to vLLM pods that already hold relevant KV-cache entries, reducing redundant prefill computation across the fleet. It also supports prefill/decode disaggregation and wide expert parallelism for large models.