Slide 1
llm-d Routing Deep Dive

How Does an LLM Request
Find Its Perfect Pod?

Inside the Endpoint Picker (EPP) — the pluggable filter → score → pick engine of the Gateway API Inference Extension that llm-d uses to route every request to the best vLLM pod.

Request → Inference Gateway → EPP: Filter → EPP: Score → Pick → vLLM Pod
3-stage
Filter → Score → Pick cycle
Live
Decisions driven by real-time vLLM metrics
KV reuse
Warm prefix hits skip redundant prefill

Markell Rawls

Slide 2

The EPP Scoring Cycle

Filter plugins first drop unhealthy or incompatible endpoints. Each remaining candidate is then scored by pluggable scorers — every scorer returns 0–1, multiplied by an operator-configured weight — and a picker selects the highest weighted sum.

score(pod) = wprefix × prefixMatch + wkv × (1 - kvCacheUtil) + wqueue × (1 - queueDepth/max) + wlora × loraAffinity
Weights are relative multipliers set per deployment — the values below are illustrative, not defaults.
💾
0.40
Prefix-Cache Affinity
⚙️
0.25
KV-Cache Utilization
📊
0.25
Queue Depth
🧩
0.10
LoRA Affinity
🛡️
✓/✗
Health & capability — a filter, not a scorer
Slide 3

The EPP Decision Pipeline

From request arrival at the gateway to pod selection — the EPP runs this cycle for every request, adding only milliseconds of overhead. Scores below use the illustrative weights from the previous slide.

1Gateway → EPP (ext-proc)
2Filter Endpoints
3Prefix-Cache Score
4KV-Util Score
5Queue Score
6LoRA + Weighted Sum
7Pick & Route
Pod A
KV cache util: 45%
Pod B
KV cache util: 82%
Pod C
KV cache util: 31%
Pod D
KV cache util: 67% · unready
Pod E
KV cache util: 55%
Pod F
KV cache util: 20%
Slide 4

Prefix & KV-Cache-Aware Routing

Prefix-cache affinity typically carries the largest scorer weight: on a warm hit, vLLM reuses the KV cache for every cached token and only prefills the remainder of the prompt.

Prefix Match Tiers (illustrative match ratios)

Full Prefix Hit
Entire prompt prefix already cached — only new tokens are prefilled
>70%
Partial Hit
Longest shared prefix reused, the rest is prefilled
20-70%
Miss
No shared prefix, full prefill required
<20%

How the EPP knows what's cached

Approximate: the prefix-cache scorer remembers which pod it routed each prefix to. Precise: vLLM emits KV-cache events that llm-d's KV-cache manager indexes for block-level lookups.

Pod Cache Inventory

Pod A: system + chat
Pod B: system + code
Pod C: system only
Pod D: empty

Prefix Tree Visualization

Slide 5

Algorithm Comparison Race

100 requests head-to-head. Watch how cache-aware scoring beats classic load balancing. This is an illustrative in-browser simulation, not a benchmark.

Round Robin
Avg TTFT--
Cache Hit--
Completed--
Re-prefill--
Least Connections
Avg TTFT--
Cache Hit--
Completed--
Re-prefill--
Random
Avg TTFT--
Cache Hit--
Completed--
Re-prefill--
llm-d EPP
Avg TTFT--
Cache Hit--
Completed--
Re-prefill--
Slide 6

Disaggregation & Priority

Scoring picks the best pod — but llm-d's scheduler also decides which kind of pod serves each phase of a request, and whether low-priority work should run at all under load.

Prefill / Decode Disaggregation

Prefill workers — compute-bound
Process the whole prompt in parallel and build its KV cache
Decode workers — memory-bandwidth-bound
Generate output tokens one step at a time from the transferred KV cache
A per-request decision
Long prompts take the disaggregated path, streaming KV blocks from prefill to decode pod over a fast interconnect; short prompts stay on one pod to avoid transfer overhead

Priority-Based Flow Control

Workloads declare priority
The Inference Extension lets each workload state its objective — latency-critical serving vs. sheddable batch traffic
Saturation detection
Queue depth and KV-cache utilization from live vLLM metrics signal when the pool is running hot
Shed low priority first
Under saturation, sheddable requests are rejected or held back so critical traffic keeps its latency SLOs
Slide 7

Weight Tuning Guide

Every workload has a sweet spot. Tune scorer weights to match your traffic pattern — the ratios below are illustrative starting points.

Workload Profile Prefix Cache KV Util Queue LoRA
Chatbot 0.60 0.15 0.15 0.10
Code Completion 0.55 0.20 0.20 0.05
Batch Processing 0.15 0.45 0.35 0.05
Multi-tenant LoRA 0.30 0.15 0.15 0.40
RAG Pipeline 0.50 0.25 0.20 0.05
Low-Latency API 0.25 0.30 0.40 0.05

Key Tuning Principles

  • Conversational workloads benefit most from high prefix-cache weight (session affinity emerges naturally)
  • Batch workloads should prioritize KV utilization and queue depth to spread load across the fleet
  • Multi-tenant LoRA deployments need adapter affinity boosted so requests land on pods with the right adapter loaded
  • Weights are relative multipliers — only their ratios matter, they don't need to sum to anything. Start from the shipped defaults and adjust one scorer at a time
  • Monitor TTFT p95 and prefix-cache hit rate as your primary tuning feedback signals
Slide 8

Intelligent Routing at Every Scale

From a single node to thousands of GPUs, llm-d's inference scheduler helps every request find the fastest path to completion.

💾

Cache-Aware

Routes to pods with warm KV cache, eliminating redundant prefill computation.

⚖️

Load-Balanced

KV-cache utilization and queue depth from live vLLM metrics prevent hotspots while maximizing throughput.

🧩

Adapter-Aware

LoRA adapter affinity ensures multi-tenant requests avoid cold-loading adapters.

llm-d.ai GitHub Docs

Markell Rawls