llm-d Open Source Framework
Your Agent Makes 7 LLM Calls.
Why Pay for 7 Cold Starts?
llm-d uses KV cache-aware routing to eliminate redundant prefills — up to 84% faster time-to-first-token in the illustrative agentic scenario below.
Every Call Pays the Full Prefill Cost
Agentic workflows invoke the LLM multiple times per query. Without smart routing, every call recomputes the KV cache from scratch, wasting GPU time on identical system prompts.
Why This Hurts
- 2,300 prompt tokens (system + tools) re-parsed every call
- 350ms cold TTFT on each of 7 calls
- Same prefix recomputed 7 times per task
- GPU compute wasted on identical prefills
KV Cache-Aware Routing with EPP Scoring
The Inference Gateway's Endpoint Picker (EPP) hashes the prompt prefix and routes subsequent calls to vLLM pods that already hold a warm KV cache. Near-zero prefill for calls 2 through 7.
into prefix key 0xa3f7c2e1
+ 20% GPU load
Skip 92% of prefill
Agent Execution Trace
Watch a 7-call agent workflow (plus two external tool executions) route through llm-d. Green bars are cache hits, amber partial matches, red cold misses, violet external tools. Timings are simulated.
The Numbers Tell the Story
Random routing vs. EPP cache-aware routing on the same 7-step agentic workflow. Illustrative simulation values — not a formal benchmark.
Cross-Agent Cache Sharing
Agents sharing the same system prompt route to pods with warm KV cache. EPP scores prefix hash matches across all active agents, enabling cross-agent cache reuse.
Cost Savings at Scale
At production scale, cache-aware routing compounds into large GPU-time savings. Illustrative cost model: 7 LLM calls per agent task, 2,300 prompt tokens per call, ~60% prefill cost avoided, 70B-class model at $0.90/M input tokens.
| Scale | Without llm-d | With llm-d | Monthly Savings |
|---|---|---|---|
| 1K tasks/day | $435/mo | $174/mo | $261/mo |
| 10K tasks/day | $4,347/mo | $1,739/mo | $2,608/mo |
| 100K tasks/day | $43,470/mo | $17,388/mo | $26,082/mo |
| 1M tasks/day | $434,700/mo | $173,880/mo | $260,820/mo |
Stop Paying for Cold Prefills
llm-d is open source, Kubernetes-native, and ready for production. Route your agentic AI workloads through cache-aware scheduling today.
llm-d is an open source, Kubernetes-native distributed inference serving stack built on vLLM.