Measured impact of prefix-aware scheduling on time-to-first-token latency across scaling, concurrency, and agentic workloads
Faster Time-to-First-Token at 32K Context Length
Controlled single-variable comparison isolating the effect of KV-cache-aware routing on inference latency.
3x NVIDIA H200 141GB SXM GPUs, tensor-parallel across nodes
Red Hat OpenShift 4.21 with llm-d operator orchestration
vLLM v0.8.5 with prefix caching enabled, 95% GPU memory
TTFT via OpenAI-compatible streaming API, first-token timestamp delta
Standard Kubernetes service load balancing -- requests distributed without cache awareness
llm-d EPP routes requests to GPU nodes where matching KV-cache prefix blocks already reside
Only the routing strategy changes -- identical model, hardware, vLLM config, and prompt content
Cold-start (round-robin) latency grows linearly with context, while cache-aware routing stays nearly flat.
At 32K tokens, round-robin takes 3,742ms while cache-aware completes in just 79ms -- a 47.5x improvement. The gap widens as context grows.
Long-context workloads (RAG, document QA, code analysis) benefit most from cache-aware routing because their large prefill is the dominant latency cost.
Ratio of round-robin TTFT to cache-aware TTFT. Higher bars indicate greater advantage from prefix caching.
How routing strategies behave under concurrent sessions and the spread of observed latencies.
| Metric | Round-Robin | Cache-Aware |
|---|---|---|
| Mean | 220.1 ms | 55.5 ms |
| Median | 234.8 ms | 46.3 ms |
| P95 | 288.3 ms | 73.0 ms |
| Min | 84.9 ms | 41.7 ms |
| Max | 289.0 ms | 73.2 ms |
Cache-aware routing maintains consistent low latency even as concurrent sessions increase. Round-robin shows high variance due to random cache misses across replicas.
How cache-aware routing performs in real agentic workflows and scales across GPU fleets.
Cache-aware routing provides stable ~70ms TTFT in 12-step agentic workflows vs. round-robin's initial 679ms spike. Multi-turn conversations benefit from prefix reuse.
Cache-aware latency stays constant at ~3.74ms (32K) regardless of fleet size, while round-robin scales linearly. At 200 replicas: 748ms vs 3.74ms.
Every LLM request must compute KV-cache for all input tokens before generating the first output token. Longer context means longer wait.
vLLM stores computed KV-cache blocks in GPU memory. If a new request shares a prefix with a cached one, it skips recomputation entirely.
Standard Kubernetes load balancing distributes requests randomly, so each GPU computes the same prefix independently -- wasting GPU cycles.
The Endpoint Picker (EPP) hashes prompt prefixes and routes to the GPU that already holds matching KV-cache blocks, achieving near-instant prefill.
Markell Rawls | AI Evangelist, Red Hat
Data collected June 2025 | llm-d open source project