Routing Algorithm Deep Dive: How llm-d's EPP Decides Where Every Request Goes

1. Why Routing Matters for LLM Inference #

In normal web-services land, routing is basically a solved problem. You've got a pool of stateless app servers, all interchangeable. Computing a shopping cart total gives the same answer at the same speed no matter which server does it. Round robin works. Least connections works. Honestly, even random works fine. The routing decision barely matters for performance.

LLM inference throws all that out the window. Every pod running an engine like vLLM keeps a KV cache -- a data structure in GPU memory holding the intermediate attention computations for token sequences it's already processed. This cache is the single biggest factor in inference latency. If a request comes in whose input tokens match a prefix that's already cached on some pod, that pod skips the expensive prefill phase for the matched part and starts generating output almost immediately. Send the exact same request to a pod with a cold cache and it has to do the whole prefill from scratch -- potentially seconds of GPU work for long contexts.

This is the problem llm-d exists to solve. At the center of it is the Endpoint Picker Policy (EPP) -- a routing component that actually understands vLLM's cache state and optimizes for compute reduction, not just spreading load around. It scores every available pod on five dimensions, combines them into one composite score, and sends each request to whichever pod will serve it fastest. This post is my full walkthrough of how that algorithm works.

2. The EPP Scoring Algorithm in Full Detail #

The EPP is the routing brain of llm-d. For every incoming request, it computes a score for every available pod and picks the highest scorer. The formula is just a weighted linear combination of five factors, each normalized to [0, 1]:

Since each factor is normalized to [0, 1] and the weights sum to 1.0 (0.60 + 0.20 + 0.10 + 0.05 + 0.05 = 1.00), the composite score is guaranteed to land in [0, 1] too. A score of 1.0 is the theoretical best case: a fully healthy pod with 100% cache match, zero GPU load, zero queue depth, and the requested adapter already loaded. Let's go through each factor.

cacheMatchRatio (weight: 0.60) #

This is just: how many token blocks of the request's input are already cached on this pod, divided by the total blocks in the input. 1.0 means the pod has the entire input prefix cached -- the request skips prefill completely and starts generating after only the cache lookup overhead (about 45ms for a 4096-token context on A100). 0.0 means a total cache miss -- the pod has to chew through the full input from scratch.

At 0.60, cache match is the dominant factor, and for good reason. Prefill is the most expensive part of inference. For a 4096-token input on Llama-3.1-70B at BF16, prefill takes about 8.2 seconds on a single NVIDIA A100 (80GB SXM). For Llama-3.1-8B at the same context length, it's about 0.9 seconds. A full cache hit wipes that out entirely. Nothing else -- not load balancing, not queue management -- comes close to that kind of latency win. The 0.60 weight means a pod with a strong cache match almost always wins the routing decision, unless it's badly overloaded or unhealthy.

loadRatio (weight: 0.20) #

loadRatio is the pod's current GPU utilization, normalized as currentUtilization / maxUtilization. The score uses the inverse, (1 - loadRatio), so low utilization scores high. A pod at 90% GPU utilization gets a load score of 0.10; an idle pod gets 1.0.

The 0.20 weight is a deliberate choice: load matters, but it shouldn't override cache affinity. Take a pod at 80% GPU utilization with a 100% cache match versus an idle pod with no cache. The cache-hot pod scores 0.60 * 1.0 + 0.20 * 0.2 = 0.64 on these two factors alone; the idle pod scores 0.60 * 0.0 + 0.20 * 1.0 = 0.20. The cache-hot pod wins by a mile even though it's way more loaded. And that's the right call -- sending the request to the idle pod would burn 8.2 seconds of GPU prefill on a 70B model for something that could've been served in 45ms from cache.

queueDepth (weight: 0.10) #

The queue score counts how many requests are already waiting in the pod's inference queue, normalized by a configurable max: 1 - min(queueDepth / maxQueue, 1). Zero queued requests scores 1.0. At or above the max, it scores 0.0. It drops linearly at 1/maxQueue per queued request -- with the default maxQueue=20, every extra queued request costs exactly 0.05.

Queue depth gets a lower weight than load because it's a jumpier signal -- queues can spike and drain within seconds. But it's an important tiebreaker: when two pods have similar cache matches, the one with the shorter queue serves the request sooner. The min(..., 1) clamp floors the score at 0.0 once a pod's queue hits maxQueue, so it can never go negative and drag down the composite.

healthScore (weight: 0.05) #

Instead of the usual binary healthy/unhealthy check, llm-d uses a continuous health score in [0, 1]. A fully healthy pod is 1.0. A pod that's partially degraded -- intermittent GPU ECC errors, elevated memory pressure, that kind of thing -- might report 0.7. Anything with healthScore < 0.3 gets excluded from routing entirely (the EPP treats its composite score as 0, so it drops out of the candidate pool). Pods in the [0.3, 1.0] range contribute their health score proportionally.

The 0.05 weight is small on purpose -- in steady state most pods are healthy, so this factor doesn't differentiate anything. It only kicks in when a pod is degraded, and then the score reduction shifts traffic away gradually instead of dropping it all at once. The maximum health penalty is 0.05 * (1.0 - 0.3) = 0.035 points (for a pod right at the exclusion threshold) -- small enough to not override a strong cache match, big enough to break ties between otherwise-equal pods.

adapterMatch (weight: 0.05) #

If you're running LoRA (Low-Rank Adaptation) fine-tuned adapters, this factor is 1.0 when the requested adapter is already loaded on the pod, 0.0 otherwise. Loading an adapter isn't free: rank-16 adapters (the common case) load in about 200ms on A100, rank-64 in about 800ms. Those numbers depend on your config -- adapter size and GPU memory bandwidth both matter.

0.05 makes sense here because adapter load time (200-800ms) is real but small next to a full cache miss (0.9-8.2 seconds depending on model size and context length). In practice it's a tiebreaker when several pods have similar cache and load scores. If you're multi-tenant with 10+ LoRA adapters, bumping w_adapter to 0.15-0.20 can be worth it -- more on that in Section 13.

3. Mathematics of Scoring #

Section 2 was the intuitive version. Now let's get formal -- notation, a couple of proofs, and a sensitivity analysis showing exactly how much pull each input variable has on the routing decision. Feel free to skim this one if math isn't your thing; the takeaway box at the bottom has the punchline.

Formal Definition #

First, the five input variables and the composite scoring function, defined precisely. Lowercase for variables, bold for vectors.

Proof: The Score is Bounded in [0, 1] #

Quick proof that for all valid inputs, the composite score S stays in the closed interval [0, 1].

Partial Derivatives (The Gradient) #

The gradient of S tells you exactly how much the composite score moves per unit change in each input. Since S is linear in each normalized factor (with a piecewise-linear clamp on the queue term), the partial derivatives are constant within each region.

Sensitivity Analysis #

So which input actually moves the score the most? Compare the partial derivatives -- but normalize by each variable's range first, or the comparison isn't fair. The "sensitivity" of S to each variable is the max score change it can cause swinging from its worst value to its best.

The numbers back up the intuition: cache match ratio is 3x more influential than load ratio, 6x more than queue depth, and 12x more than health or adapter match. A pod going from 0% to 100% cache match gains 0.60 points. To get that same 0.60 from load alone, load ratio would have to drop by 3.0 -- but it's capped at [0, 1], so the most load can ever give you is 0.20 points. Cache dominance isn't a coincidence, it falls straight out of the weight configuration.

Proof: Cache Match Dominance #

There's a stronger result here. Let's try to prove that with default weights, a pod with a cache match advantage of at least 0.40 (40 percentage points) always outscores a pod with zero cache match, no matter what the other variables do. (Spoiler: the first attempt fails in an interesting way.)

This one's genuinely useful to know operationally. Once a pod has a 65%+ cache match advantage over the next-best option, no combination of load, queue, health, or adapter factors can flip the routing decision. And in practice, multi-turn conversations routinely build up cache advantages of 80-90% (see Section 9, Example 4) -- at that point the conversation is basically welded to its pod.

Worked Derivative Example: d(Score)/d(cacheMatch) #

Here's a concrete run-through of the partial derivative in action. The question: if cache match improves from 50% to 60%, how much does the score change?

Cross-Factor Equivalence Table #

This table shows how much change in each variable equals a 10-percentage-point bump in cache match. Handy when you're reasoning about tradeoffs like "is it worth sending this to a pod with 10% less cache if its queue is X shorter?"

4. KV Cache Prefix Matching -- How Prefix Trees Work #

So cache match ratio is the most important factor in the formula -- but actually computing it efficiently across a fleet of pods is a real distributed systems problem. Here's how llm-d pulls it off.

Tokenization and Block Hashing #

Every request starts as a text prompt. The engine tokenizes it into token IDs with the model's tokenizer, then groups the tokens into fixed-size blocks -- typically 16 tokens each (configurable via vLLM's block_size). Each 16-token block gets hashed with xxHash64 into a 64-bit fingerprint. Collision odds (two different 16-token blocks producing the same hash) are about 1 in 2^64, roughly 5.4 x 10^-20 -- so, not something you'll ever see.

Why blocks instead of individual tokens? Tradeoff. Token-level matching would be more precise but would blow the prefix tree up to millions of nodes on a busy cluster and slow every lookup. Block-level matching shrinks the tree 16x and costs you basically nothing -- vLLM evicts at block granularity anyway, so finer-grained matching in the EPP wouldn't buy you anything. For a 4096-token input, the lookup walks exactly 256 nodes (4096 / 16), which finishes in under 50 microseconds on modern hardware.

The Prefix Tree Structure #

The EPP keeps a prefix tree (radix tree, trie, whatever you want to call it) mapping block hash sequences to the pods that have them cached. It's a single in-memory structure inside the EPP process. Each node is one block hash; a path from the root to any node is a cached prefix -- a contiguous run of block hashes matching a token prefix stored on one or more pods.

Each node carries a set of pod IDs -- which pods have this exact prefix cached. Interior nodes are partial prefixes; leaf nodes (or the longest path matching your query) are the deepest cache match available.

How the EPP Maintains a Global View #

Each pod periodically reports its cache state to the EPP through lightweight status updates -- basically "here are the block hashes currently sitting in my GPU memory." The EPP folds these into the tree: new paths when a pod caches new prefixes, removals when a pod evicts. It's eventually consistent, which means there's a window (typically 1-5 seconds, set by the pod's reporting interval) between a pod evicting an entry and the EPP finding out. I get into what that race condition means in Section 15.

How Lookup Works #

When a request comes in, the EPP hashes its input into blocks and walks the tree from the root. At each level it looks up the next block's hash: matching child node, record which pods hold it and descend; no match, stop. What you end up with is, per pod, the count of consecutive blocks matched from the start of the input.

The match ratio for each pod is then:

A ratio of 0.75 means the pod has the first 75% of the input cached. For a 4096-token input on Llama-3.1-70B, that means skipping prefill on 3072 tokens (about 6.15 seconds of compute) and only prefilling the remaining 1024 (about 2.05 seconds). A ratio of 1.0 means zero prefill -- just the 45ms lookup overhead -- which is the ideal case and the whole point of the routing system.

Partial vs. Full Matches #

Partial matches are common and they're worth a lot. In a chatbot, the first message establishes the system prompt prefix. The second message shares the system prompt plus the first exchange. A pod that cached turn one has a partial match for turn two -- the system prompt and first exchange are cached, only the new user message needs prefill. That partial match might cover 80-90% of the input, and you save that fraction of the prefill compute.

Full matches (ratio = 1.0) happen when the exact same input has been seen before -- common for system prompts, shared document prefixes, and repeated queries. Prefill drops to about 45ms on A100, limited only by cache lookup and attention recomputation overhead.

5. Why Traditional Load Balancing Algorithms Fail for LLM Inference #

To see why llm-d needs a custom routing algorithm, it helps to see why every standard one falls over. And these aren't subtle failures -- they're structural.

Round Robin #

Round robin cycles requests through pods in order: pod 1, pod 2, pod 3, repeat. It assumes pods are interchangeable -- that a request runs equally well anywhere. For LLM inference that's just false. Round robin ignores cache state completely, so a request with a perfect cache match on pod 1 might get sent to pod 3's cold cache because, well, it's pod 3's turn. You get pointless prefill, wasted GPU cycles, higher latency. At scale the cache miss rate approaches 100%, because scattering requests uniformly means no pod ever builds a useful cache for any prefix.

Least Connections #

Least connections sends each request to the pod with the fewest active connections. Better than round robin since it at least sees load -- but it actively fights cache locality. A pod that's been serving a popular system prompt has a warm cache and, naturally, more active connections than a freshly started cold pod. So least connections routes new requests -- including ones that would've hit that warm cache -- to the cold pod, where they eat the full prefill cost. It optimizes load distribution at the direct expense of compute efficiency.

Random #

Random is the worst case for cache locality. The odds a request lands on a pod with useful cached state are 1/N for N pods -- in a 32-pod cluster, 97% of requests hit a cold cache. Every pod becomes an island with its own fragmentary cache, and none has enough memory to hold a useful working set.

Weighted Round Robin #

Weighted round robin can handle heterogeneous hardware -- more traffic to beefier GPUs. Still cache-blind though. A pod with an A100 and an empty cache is worse for a given request than a pod with an A10 and a warm cache for that request's prefix, because the cache hit skips prefill entirely.

It all comes back to the same point: for LLM inference, WHERE a request runs determines HOW FAST it runs. On a cache-hot pod, prefill overhead is about 45ms. On a cold pod, the same request takes 0.9-8.2 seconds of prefill depending on model size. No traditional load balancer models this because no traditional workload has it -- web servers, databases, and microservices are either stateless (same computation everywhere) or their state lives in shared storage everyone can reach. The KV cache is neither. It's pod-local, volatile, and it directly sets your execution cost.

6. Queue Depth Scoring -- Why It Matters More Than GPU Utilization #

GPU utilization is the metric everyone watches, and it's the wrong one for routing. Utilization is a lagging indicator -- it tells you what the GPU was doing over the last sampling interval (typically 100ms-1s, depending on your NVIDIA DCGM or nvml polling config). By the time it spikes to 95%, the damage is done: the GPU is saturated and requests are already queuing. You can't route around a problem you only detect after it happened.

Queue depth is a leading indicator. It's about future latency, not past utilization. Queue depth of 10 means your next request waits behind 10 others before it even starts. Even if GPU utilization happens to read 40% right now (maybe a batch just finished and the next one hasn't started), those 10 queued requests will keep that GPU busy for a while.

The Counterintuitive Case #

Here's the case that trips people up. Two pods:

• Pod A: 80% GPU utilization, 0 queued requests
• Pod B: 40% GPU utilization, 10 queued requests

A traditional load balancer picks Pod B -- lower utilization, easy call. But Pod A is actually the better choice. Its GPU is busy but the queue is empty, so your request starts as soon as the current batch finishes. Pod B looks less loaded right now, but your request sits behind 10 others. By the time it starts processing, Pod A would've finished long ago.

Preventing Pile-up on Cache-Hot Pods #

Queue depth scoring is what keeps the EPP from drowning popular pods. Without it, a pod with a warm cache for a popular system prompt would get every single request for that prompt, its queue would grow forever, and latency would blow up. The queue factor is backpressure: as the queue grows the pod's score drops, and eventually a less-cached but less-loaded pod becomes the better pick. That request pays some prefill cost, but skips the queue wait.

Queue Normalization and the Linear Gradient #

The queue score is:

maxQueue is configurable (default: 20) -- it's the queue depth where the pod counts as fully saturated. The score is linear in queue depth with slope -1/maxQueue. With maxQueue=20:

The min(..., 1) clamp floors the score at 0.0 past maxQueue. The weighted contribution is w_queue * queueScore = 0.10 * queueScore, so the biggest possible queue penalty is 0.10 points (0.10 at empty queue down to 0.00 at full). Deliberately much smaller than cache's max of 0.60, but enough to push traffic off badly congested pods.

Graceful Degradation Under Congestion #

When every pod is congested (all queues deep), queue scores converge toward zero across the board and cache match becomes even more dominant. Which is exactly right: if your request is going to sit in a queue no matter what, you might as well send it to the pod that'll process it fastest when it reaches the front -- the one with the best cache match. The system degrades gracefully instead of making arbitrary calls under pressure.

7. Health Checking -- Beyond Simple Liveness #

Kubernetes liveness and readiness probes answer yes/no questions: is the process alive? Can it take traffic? For LLM inference, that's necessary but nowhere near enough. A pod can be "live" and "ready" by Kubernetes standards and still be a terrible place to send an inference request.

What llm-d Health Checks Include #

• Is the model loaded? Loading a 70B parameter model from storage into GPU memory takes 120-300 seconds depending on storage bandwidth (NVMe vs. network-attached). During this time, the pod is alive and its HTTP server is responding to health checks, but it cannot serve inference requests. The llm-d health check distinguishes between "process running" and "model loaded and ready for inference."
• Is the GPU healthy? GPUs can experience ECC (Error-Correcting Code) memory errors, thermal throttling, and memory corruption without crashing the process. A pod with accumulating ECC errors may produce correct results today but is at elevated risk of failure. Thermal throttling (triggered above 83C on A100) reduces compute throughput by 10-30% without any visible error. The EPP health check incorporates GPU health metrics from NVIDIA's management libraries (NVML).
• Is the cache warm? A freshly restarted pod has an empty KV cache. It is technically healthy and ready for inference, but every request will require full prefill. The EPP treats cache warmth as a health signal -- a cold pod is not "unhealthy," but it is less preferred than a warm pod for most requests.
• Is the inference engine responsive? The vLLM process can enter states where it is running but not processing requests -- for example, during garbage collection pauses, CUDA synchronization stalls, or internal queue deadlocks. The EPP performs active inference-level health checks (not just TCP connection checks) to detect these conditions.

Continuous Health Scores and Exclusion Thresholds #

The health score is a continuous value in [0, 1], not a flag, and that opens up more nuanced routing:

• 1.0: Fully healthy -- model loaded, GPU healthy, cache warm, engine responsive.
• 0.7-0.9: Minor degradation -- intermittent ECC errors, moderate thermal throttling, or elevated memory pressure. The pod still receives traffic but at reduced priority.
• 0.3-0.6: Significant degradation -- the pod is functional but should only receive traffic when better alternatives are unavailable.
• < 0.3: Below exclusion threshold -- the EPP zeroes out this pod's composite score, effectively removing it from the candidate pool. The pod receives no new traffic. Existing in-flight requests complete normally.
• 0.0: Draining or failed -- the pod is explicitly marked as unavailable.

Why does continuous beat binary here? Because routing decisions are expensive to get wrong. Yank a pod from the pool abruptly and its cached state is gone, instantly. Winding traffic down gradually gives the system time to replicate cached prefixes onto other pods, so the disruption stays small.

8. Session Affinity -- Multi-Turn Conversation Routing #

Multi-turn conversations are the bread and butter of real LLM traffic. Chatbots, coding assistant sessions, interactive document analysis -- they're all a series of requests where each turn builds on the last. Every turn appends tokens: system prompt, first user message, first assistant response, second user message, and so on. By turn five the input can be thousands of tokens.

Cache-Aware Routing Provides Natural Session Affinity #

My favorite property of the whole algorithm: session affinity just falls out of it. Nobody configures it. Here's why:

After turn one, the pod that served it has the system prompt plus that first exchange cached. Turn two arrives with a prefix of system prompt + first exchange + new user message. The prefix tree lookup finds that the turn-one pod has the longest match (the entire prior context), so it scores highest on cache and almost certainly wins turn two. Same logic for turn three, four, and on. Each turn extends the cached prefix on the same pod, and the scoring keeps sending the conversation back there.

When the Preferred Pod is Overloaded #

If the pod that's been serving the conversation is now slammed (deep queue, high load), the EPP may route a turn elsewhere. The new pod usually has a partial match -- maybe the shared system prompt is cached but the conversation-specific context isn't -- so it does a partial prefill for the uncached part. That's the cost of migration: a one-time prefill penalty for whatever context only lived on the original pod.

Still strictly better than traditional affinity, which would either (a) park the request behind everything else on the overloaded pod, or (b) fail over to a totally cold pod. The EPP finds the middle: route to the best available pod -- often a partial match -- and pay a proportional prefill cost.

Comparison to Traditional Session Affinity #

Cookie-based affinity maps a session ID to a pod -- scale that pod down and the session breaks; crash it and the state's gone. IP hashing falls apart behind NAT and with mobile users whose IPs change constantly. Header routing needs the application to cooperate. None of these have any idea what the pod actually has cached for this session. They're all indirect proxies for the signal the EPP measures directly: cache content.

9. The Mathematics -- Worked Examples #

Formulas are one thing; let's actually run the numbers. Five full examples, all using the default weights: w_cache=0.60, w_load=0.20, w_queue=0.10, w_health=0.05, w_adapter=0.05 and maxQueue=20.

10. Comparison with Other Routing Approaches #

A fair question at this point: couldn't you do this with the proxies everyone already runs? I went down that road, so here's the comparison.

Envoy Proxy #

Envoy is a very capable L7 proxy -- header-based routing, weighted clusters, circuit breaking, retries. But it lives entirely at the HTTP level. It has no idea what a KV cache is, no way to query vLLM's internal state, no prefix matching. You can set up consistent hashing on a header or URL, but the hash key is HTTP metadata, not token content. Two requests with identical prompts but different headers hash differently; two requests with different prompts but the same URL hash the same. The hash key is just wrong for LLM routing.

Istio #

Istio puts a control plane on top of Envoy -- VirtualService, DestinationRule, weight-based splitting, header routing, fault injection. But its routing rules are static or weight-based: they describe how traffic should be distributed, not where each individual request should go. There's no way for Istio to check each pod's cache state at request time and route on it. Istio manages traffic policy; the EPP solves a lower-level, per-request placement problem. Different jobs.

NGINX #

NGINX's hash directive is the closest thing to cache-aware routing among the traditional options. Hash on the request URI or a custom header, get consistent pod assignment for identical keys, and you do get a form of cache locality -- all requests for the same system prompt land on the same pod, which builds a warm cache. The problem, again, is the hash key. NGINX hashes the URL or a header, not the tokenized content. Tiny variations -- formatting, extra whitespace, an added parameter -- produce different hashes and different pods even when the token prefix is identical. And consistent hashing has no notion of partial matches: a request either hashes to the "right" pod or it doesn't. There's no "this pod has 60% of the prefix cached."

HAProxy #

HAProxy is genuinely good at health checking and queue management -- per-server connection limits, queuing of excess requests, active probes, weighted health. Of all the traditional load balancers, its health checking is the closest to the EPP's. But it has no inference-specific metrics. It can't query a pod's KV cache, doesn't understand token prefixes, can't factor in adapter loading. Great general-purpose load balancer; blind to the things that matter most here.

11. Gateway API Integration #

The EPP doesn't run in a vacuum -- in a real deployment it plugs into the Kubernetes Gateway API, the standard that's replacing the old Ingress resource. If you want to deploy llm-d properly you need to understand how the EPP fits into that model, so here's the architecture, the custom resources, and the full request flow from client to pod.

The InferencePool and InferenceModel CRDs #

llm-d extends the Gateway API with two CRDs that describe the inference topology:

InferencePool is a set of pods running the same base model behind a shared EPP -- the basic unit of capacity in llm-d. It specifies which model is loaded, the resource requirements (GPU type, memory), and the EPP config (scoring weights, maxQueue, reporting intervals). Basically a deployment group that also carries inference-specific metadata.

InferenceModel is the logical model name clients put in their API requests. It maps client-facing model identifiers to the InferencePool(s) that serve them, and declares which LoRA adapters are available. Multiple InferenceModels can point at the same pool -- that's how you serve several fine-tuned variants off one set of pods.

How HTTPRoute Connects to the EPP #

The standard HTTPRoute resource says which requests go to which backend. In llm-d, the HTTPRoute points at an InferencePool instead of a Kubernetes Service. The Gateway controller takes the request, matches it against the HTTPRoute rules, and hands it to the EPP for that pool -- which then makes the pod-level call using the scoring algorithm from this post.

Full Request Flow #

The full path of a request through the llm-d stack touches five components. Here's the flow, annotated:

The EPP runs as an external processing filter (ext_proc) -- a gRPC service the Gateway calls mid-request to make the routing decision. That's a standard extension mechanism in Envoy-based gateways. Importantly, the EPP is not in the data path for the response: after the routing decision, the response streams straight from the pod back through the Gateway to the client. So the EPP adds sub-millisecond latency for the scoring computation and never becomes a throughput bottleneck.

Comparison with Traditional Ingress-Based Routing #

Why not plain old Ingress? It does basic HTTP routing -- host and path matching, TLS termination, backend references to Services -- but for LLM inference it has some hard limits:

• No extension mechanism: Ingress has no concept of ext_proc or external routing filters. You cannot plug the EPP into an Ingress controller. The routing decision is hardcoded to the Ingress controller's built-in algorithms (typically round robin or IP hash).
• Service-level granularity: Ingress routes to a Kubernetes Service, which then uses kube-proxy (iptables or IPVS) to select a pod. There is no way to influence the pod selection from outside. The EPP needs to select specific pods, which requires the Gateway API's backend-level routing.
• No typed backend references: Ingress can only reference Services. Gateway API can reference arbitrary backends (including InferencePool), enabling the EPP integration pattern.
• Limited extensibility: Ingress annotations are implementation-specific and fragile. Gateway API provides a standardized extension model through policy attachment and custom backend types.

Multi-Model Routing via InferenceModel Selectors #

A really common pattern: serving multiple fine-tuned models from one InferencePool. Say a customer support platform offers "customer-support-v3" (general), "customer-support-v3-enterprise" (tuned for enterprise clients), and "customer-support-v3-billing" (tuned for billing questions). All three share the same base model, so they can all run on the same pods with different LoRA adapters.

Each variant gets its own InferenceModel, all pointing at the same pool with different targetModels. A request for "customer-support-v3-billing" comes in, the EPP resolves the InferenceModel, sees it needs the "billing-lora-v3" adapter, and folds the adapter match into the score. Pods that already have the billing adapter loaded get the 0.05-point bonus (default weights), which breaks ties when cache and load are close.

12. Custom Scoring Plugins #

The default five-factor formula covers most routing needs, but sometimes you need scoring logic it just can't express. Maybe you want to factor in per-pod cost and route to cheaper spot instances when latency allows. Maybe your research cluster should prefer certain GPU architectures for certain model configs. Maybe you have data residency rules and need to penalize pods in the wrong availability zone.

Good news: the scoring pipeline is pluggable. You write a custom scorer in Go, register it with the EPP, and it participates in the calculation right alongside the built-in factors. Here's the architecture, and then we'll build one.

Architecture of the Scorer Interface #

Every scoring factor implements the same Scorer interface. A scorer gets the request context and a candidate pod, and returns a normalized score in [0, 1] plus metadata for logging and debugging.

InferenceRequest carries the tokenized input, the requested model name, adapter requirements, and request metadata (headers, timestamps, client identity). PodState carries the pod's current metrics: cache contents, GPU utilization, queue depth, health score, loaded adapters, and any custom annotations the pod reports.

The Scoring Pipeline #

On each request, the EPP runs all registered scorers in parallel for every candidate pod, then collects, weights, and sums the results into the composite score. The pipeline looks like this:

Writing a Custom Scorer: Cost Awareness #

Let's build one: a "cost awareness" scorer. The setup -- you're running a mixed GPU cluster with on-demand A100s and spot-priced A10Gs. On-demand A100s run $3.67/hour, spot A10Gs run $0.38/hour. You want the EPP to lean toward the cheap instances when the latency hit is acceptable -- specifically, when the cache match gap between the expensive pod and the cheap pod is under 20 percentage points.

Plugin Registration #

Custom scorers register at EPP startup. Registration validates that all weights sum to 1.0 and names are unique -- so when you add one, you have to take weight away from the defaults to keep the sum at 1.0.

When Custom Scorers Make Sense vs. Weight Tuning #

Weight tuning (Section 13) changes the relative importance of the five built-in factors, and it's enough whenever your need boils down to "more cache" or "less queue sensitivity." Custom scorers are for when you need a genuinely new signal -- something the five defaults don't capture at all.

How I'd decide:

• Use weight tuning when the signal you care about is already a factor (cache, load, queue, health, adapter) but at the wrong priority. Example: "Cache matters less for my batch workload" -- reduce w_cache.
• Use a custom scorer when the signal is not captured by any existing factor. Examples: cost, data residency, GPU architecture preference, request priority levels, client SLA tiers.
• Avoid custom scorers for signals that are highly correlated with existing factors. If your "priority" signal is really just "give this request a shorter queue wait," increase w_queue instead. Adding a redundant scorer wastes weight budget.

Testing Strategies for Custom Scorers #

Test custom scorers at three levels:

13. Tuning Guide: When to Adjust Which Weight #

The default weights (0.60 / 0.20 / 0.10 / 0.05 / 0.05) are tuned for general-purpose inference with moderate-to-high prefix reuse. Real workloads vary a lot, and matching the weights to yours can buy you real latency and throughput. Here's what I'd actually do, by workload.

Workload-Specific Weight Recommendations #

Decision Flowchart for Weight Tuning #

Or work from symptoms -- this flow tells you which weight to touch:

14. Debugging Routing Decisions #

When requests aren't landing where you expect, you need to see exactly what the EPP decided and why. Here's the tooling for that -- per-request logs up through cluster-wide Prometheus metrics.

Tracing a Single Request Through the EPP #

Every request the EPP touches gets a unique requestID (UUID v4) that shows up on every log line for that request, so tracing one is just filtering the logs by ID. The EPP logs at three points:

1. Request received: Logged when the ext_proc filter receives the request. Includes the model name, input token count, and any adapter requirements.
2. Scoring complete: Logged after all pods have been scored. Includes the composite score and per-factor breakdown for the top 3 pods (and the selected pod if it is not in the top 3).
3. Pod selected: Logged when the routing decision is finalized. Includes the selected pod name, its IP address, and the reason for selection.

EPP Logging Configuration #

Logging is structured JSON with configurable levels. For routing debugging you care about:

• INFO (default): Logs the final routing decision for each request -- the selected pod, composite score, and cache match ratio. This is sufficient for most operational monitoring.
• DEBUG: Adds per-factor score breakdowns for all candidate pods, prefix tree lookup results, and timing information. Use this when investigating specific routing anomalies.
• TRACE: Adds block-level prefix tree walk details, individual pod status report processing, and weight calculation intermediates. Use sparingly -- this generates 10-50x the log volume of INFO.

Set the log level via the EPP_LOG_LEVEL environment variable or the --log-level command-line flag on the EPP deployment.

Key Log Fields #

These are the log fields I actually look at when debugging routing:

At DEBUG you also get the breakdown for runner-up pods, which is what you need for understanding close calls:

Prometheus Metrics #

The EPP exposes Prometheus metrics on its /metrics endpoint (default port 9090). This is your cluster-wide view for routing visibility and alerting.

Common Debugging Scenarios #

Scenario A: "Why is my request going to the wrong pod?"

Symptom: a multi-turn conversation that's been happily routing to Pod A suddenly sends a turn to Pod C, which has a cold cache for this conversation.

How I'd chase it down:

1. Find the request's requestID from the response headers or client logs.
2. Grep the EPP logs for that requestID at DEBUG level.
3. Check the candidate scores: was Pod A scored lower than Pod C?
4. If Pod A has a lower score, check its individual factors. Common causes:
   • Cache staleness: Pod A evicted the conversation's cache due to memory pressure, but the EPP has not yet received the updated cache report. Check epp_stale_cache_routing_total for recent increments.
   • Queue buildup: Pod A's queue hit maxQueue, making it ineligible. Check epp_queue_depth{pod="pod-a"}.
   • Health degradation: Pod A's health score dropped, reducing its composite score. Check epp_pod_health_score{pod="pod-a"}.
5. If Pod A was not in the candidate list at all, it may have been excluded (health below 0.3) or removed from the InferencePool.

Scenario B: "Why is latency spiking?"

Symptom: p99 TTFT went from 200ms to 2000ms over the last hour.

Checks, in order:

1. Check epp_cache_match_ratio histogram: has the distribution shifted toward lower values? A drop in cache hit rates means more prefill, which means higher TTFT.
2. Check epp_queue_depth across all pods: are queues building up? Rising queue depths indicate the cluster is under-provisioned for the current request rate.
3. Check epp_candidate_pods: has the pool shrunk? Pod failures or scale-down events reduce capacity and increase per-pod load.
4. Check epp_scoring_latency_us: is the EPP itself slowing down? This can happen if the prefix tree has grown very large (check epp_prefix_tree_nodes). Scoring latency above 5ms is a red flag.
5. Cross-reference with epp_stale_cache_routing_total: a spike in stale routing indicates the prefix tree is out of sync with actual pod state, causing expected cache hits to become misses.

Scenario C: "Why are adapters thrashing?"

Symptom: epp_adapter_loads_triggered_total is climbing fast. Pods are loading and unloading LoRA adapters multiple times a minute.

Steps:

1. Check the adapter label on epp_adapter_loads_triggered_total: is thrashing concentrated on one adapter or spread across many?
2. If concentrated: one adapter's requests are being distributed across too many pods. Increase w_adapter to consolidate requests on fewer pods.
3. If spread: too many adapters for the pool size. Each pod can realistically serve 3-5 LoRA adapters without thrashing (depending on adapter rank and GPU memory). If you have 20 adapters on 4 pods, thrashing is inevitable.
4. Check whether cache affinity is fighting adapter affinity: a pod with good cache but the wrong adapter will be selected, triggering an adapter load. If this is frequent, increase w_adapter from 0.05 to 0.15-0.20 (and decrease w_cache by the same amount).

Grafana Dashboard Recommendations #

For ongoing monitoring, these are the Grafana panels I'd build:

• Panel 1: Cache Hit Rate Distribution -- Heatmap of epp_cache_match_ratio over time. Shows whether the cluster's cache effectiveness is improving or degrading. Alert if p50 cache match drops below 0.40 for more than 10 minutes.
• Panel 2: Queue Depth per Pod -- Time series of epp_queue_depth with one line per pod. Shows load distribution. Alert if any pod stays above maxQueue * 0.8 for more than 5 minutes.
• Panel 3: Routing Decision Rate -- Counter rate of epp_routing_decisions_total, broken down by model. Shows request throughput and model popularity.
• Panel 4: Composite Score Distribution -- Histogram of epp_composite_score over time. A bimodal distribution (cluster of scores near 0.8 and another near 0.3) indicates a split between cache-hot and cache-cold routing -- which is normal. A uniform distribution near 0.3 indicates the cache is not providing value.
• Panel 5: Stale Cache Rate -- Counter rate of epp_stale_cache_routing_total divided by epp_routing_decisions_total. This is the percentage of routing decisions that were based on stale cache data. Alert if above 5%.
• Panel 6: Scoring Latency -- Histogram quantiles (p50, p99) of epp_scoring_latency_us. Should stay below 1000us (1ms) at p99. Alert if p99 exceeds 5000us.
• Panel 7: Pod Health Overview -- Table or heatmap of epp_pod_health_score for all pods. Color-code: green (1.0), yellow (0.5-0.9), red (below 0.3). Provides at-a-glance fleet health.
• Panel 8: Adapter Load Events -- Counter rate of epp_adapter_loads_triggered_total broken down by adapter name. Spikes indicate adapter thrashing.

15. Edge Cases and Failure Modes #

No routing algorithm gets to live in a clean room. Real systems hit failures, race conditions, and hostile workload patterns. Here's how the EPP behaves when things get ugly.

Best Pod is Overloaded (Queue Full) #

When the best-cache pod's queue is full (queueDepth >= maxQueue), its queue score drops to 0.0 -- which with default weights only costs 0.10 points. For a cold pod to beat it on load + queue alone, it would need to overcome the 0.60 cache advantage, which is mathematically impossible at cacheMatchRatio = 0.0. So scoring alone can't save you here, and the EPP applies a hard eligibility cutoff instead: pods at or above maxQueue get pulled from the candidate pool outright. Traffic spills to the next-best pod, often a partial match, which pays a proportional prefill cost but dodges the queue wait.

Tie-Breaking: Equal Cache Matches #

When several pods have the same cache match (common with shared system prompts), the other factors break the tie -- load first (0.20), then queue depth (0.10). So among equally-cached pods the EPP just picks the least loaded one. When cache can't differentiate, it quietly turns back into a load balancer.

Pod Crash Mid-Request #

If a pod crashes mid-request, the request fails. The EPP does not auto-retry, because streaming inference isn't idempotent -- you can't pick up a partially streamed response on a different pod. The client has to retry, and the retry gets routed to the pod with the next-best cache. If the crashed pod held unique cached state, that state is gone, and requests that would've used it pay full prefill until the cache gets rebuilt somewhere else.

Cold Start #

The first request for a brand-new prefix always pays full prefill -- there's no cache anywhere in the cluster to match. The EPP routes it on load and queue (all cache scores are 0.0, so the cache term adds nothing for anyone and the remaining weights -- load 0.20, queue 0.10, health 0.05, adapter 0.05 -- effectively renormalize among themselves). Once it completes, the serving pod's cache is populated and future requests for that prefix route there and get the hit. Cold start is unavoidable; what the EPP can do is send the cold request to the least-loaded pod and make sure the cache it creates gets reused.

Cache Eviction Race #

Remember, the prefix tree is only eventually consistent with what's actually on the pods. There's a 1-5 second window (set by the reporting interval) between a pod evicting a prefix (memory pressure, LRU) and the EPP removing it from the tree. In that window the EPP can route a request based on a cache match that no longer exists. The pod discovers the miss and does a full prefill. Annoying, not catastrophic -- the request completes, just slower than expected, and the next status update fixes the tree.

Thundering Herd #

When a pile of requests for the same prefix lands at once (Example 5 walks through this), the EPP has to spread them out instead of dumping everything on the one pod with the best cache. The queue factor and the maxQueue hard cutoff do exactly that: the first requests fill the cache-hot pod's queue, the rest spill elsewhere. Those pods do the prefill, build their own caches, and are ready for the next wave with the same prefix. The herd actually speeds up cache warming across the cluster. In a 4-pod cluster with maxQueue=20, the first 20 requests hit the cache-hot pod and requests 21-50 spread across the other 3, warming their caches in parallel.

Memory Pressure and Cache Eviction #

GPU memory is finite. An A100 80GB SXM has roughly 72GB left for KV cache once model weights are loaded (a 70B model at BF16 needs about 140GB of weights spread across tensor-parallel GPUs; per-GPU KV cache allocation depends on the parallelism setup). When a pod's KV cache fills up, vLLM evicts entries LRU-style -- oldest, least-accessed prefixes go first. The pod reports the eviction on its next status update and the EPP prunes the tree. That's just steady state in a busy cluster: popular prefixes stay cached, unpopular ones fall out, and the prefix tree ends up mirroring the cluster's aggregate working set. The EPP never manages eviction itself -- it watches and adapts.

Common Pitfalls to Avoid #

Deploying the EPP is easy; tuning it has sharp edges. These are the mistakes I see (and have made) most often with llm-d's routing layer.

• Pitfall 1: Setting weights that do not sum to 1.0

  If your five weights (cache, load, queue, health, adapter) don't sum to exactly 1.0, the score range shifts in ways you won't enjoy. Weights summing to 1.2 produce scores up to 1.2, breaking anything that assumes [0, 1]. Weights summing to 0.8 throw away 20% of the dynamic range and make it harder to tell pods apart. Check the sum after every weight change, especially when adding custom scorers.

• Pitfall 2: Setting maxQueue too low or too high

  maxQueue=5 spills too early -- requests bounce to cold pods before the cache-hot pod is anywhere near saturated, wasting GPU time on prefill that didn't need to happen. maxQueue=100 lets queues grow unchecked, so requests wait behind dozens of others and latency spikes. The default of 20 is fine for most things. Go 8-12 for latency-sensitive APIs, 30-40 only for batch workloads where throughput beats latency.

• Pitfall 3: Ignoring the pod reporting interval (stale cache data)

  The prefix tree is only as fresh as each pod's last status update. With the default 2-second interval there's a window where the EPP thinks a pod has data it already evicted. High cache churn makes this worse. Watch epp_stale_cache_routing_total, and if stale routing goes past 3-5% of decisions, drop podReportIntervalMs to 500-1000ms.

• Pitfall 4: Over-tuning weights without production traffic data

  Tuning weights off synthetic benchmarks or gut feel usually makes things worse. Synthetic traffic never looks like the real thing -- real traffic is bursty, prefix reuse is uneven, adapter popularity shifts around. Start with defaults, measure at least 24 hours under real traffic, then move one weight at a time by no more than 0.10 while watching cache hit rate, p99 TTFT, and queue depth distribution.

• Pitfall 5: Expecting the EPP to retry failed requests

  The EPP won't retry when a pod crashes or errors out. Streaming inference isn't idempotent -- you can't resume a half-streamed response on another pod. Retries belong at the client or gateway level. If you need request-level fault tolerance, have your Gateway retry on connection errors, just know the retry will probably land on a different pod with a cold cache.

• Pitfall 6: Confusing GPU utilization with queue depth

  One more time: GPU utilization is lagging, queue depth is leading. A pod at 80% utilization with an empty queue beats a pod at 40% with 10 queued requests. Don't crank w_load to "fix" routing when the real problem is queue buildup -- raise w_queue or lower maxQueue instead.

16. Conclusion and What is Next #

If there's one framing to walk away with: the EPP isn't a load balancer, it's an inference placement optimizer. It understands the economics of LLM computation and puts each request on the pod that can run it with the least compute. The five-factor formula encodes those priorities: cache affinity at 0.60 because a cache hit on a 70B model saves 8.2 seconds of prefill; load at 0.20 to keep hot pods from saturating; queue depth at 0.10 for backpressure and latency predictability; health at 0.05 for graceful degradation; adapter affinity at 0.05 to dodge the 200-800ms cost of unnecessary LoRA loads.

The architectural decisions that make it all work:

1. A global prefix tree of xxHash64 block fingerprints that gives the EPP visibility into every pod's cache state -- the content-aware routing no HTTP-level proxy can do. Lookups finish in under 50 microseconds for typical inputs.
2. Continuous scoring instead of binary thresholds, so degradation is smooth and tradeoffs are nuanced. The composite score always lands in [0, 1] and behaves predictably.
3. Configurable weights, so you can tune for your workload -- cache-heavy chatbots (w_cache=0.60), latency-sensitive API gateways (w_queue=0.30) -- without touching the underlying logic.
4. Emergent session affinity that falls out of cache-aware routing for free, replacing brittle sticky sessions. Conversations stick to their pod harder as the cached context grows (Example 4).
5. Gateway API integration through the InferencePool and InferenceModel CRDs -- Kubernetes-native, standard HTTPRoute routing, ext_proc-based pod selection.
6. A pluggable scoring pipeline, so custom scorers can add new signals (cost, data residency, SLA tiers) without forking the core algorithm.

Routing is one piece of llm-d -- there's also autoscaling, disaggregated prefill/decode, multi-model orchestration, and more. But it's the foundational piece: every other optimization depends on requests landing on the right pod in the first place.

What is Next: Disaggregated Prefill/Decode #

The thing I want to dig into next is disaggregated prefill/decode -- running the prefill and decode phases on different pods, or even different GPU types, each optimized for its phase. Prefill is compute-bound and wants high-FLOPS GPUs (H100 SXM: 989 TFLOPS BF16). Decode is memory-bandwidth-bound and wants fast memory (H100: 3.35 TB/s HBM3). Disaggregation splits the phases across specialized hardware, and the routing layer has to orchestrate the handoff: prefill on a compute-optimized pod, transfer the KV cache to a decode-optimized pod, route subsequent tokens there. Much harder routing problem, but it builds directly on the prefix-matching and cache-awareness stuff in this post. Probably a future writeup.

Further Reading #

If you want to go deeper on anything I covered:

• llm-d Source Code (GitHub) The EPP scoring pipeline, prefix tree implementation, and Gateway API integration code.
• llm-d Documentation Official documentation covering deployment, configuration, tuning, and architecture guides.
• Kubernetes Gateway API The specification for Gateway, HTTPRoute, and the extension model that llm-d builds on.
• Gateway API Concepts: API Overview Core concepts including GatewayClass, Gateway, and Routes -- essential background for the InferencePool integration.
• vLLM Documentation The inference engine that llm-d's EPP is designed to work with -- KV cache management, block size configuration, and LoRA adapter support.