Basically: every LLM deployment makes a routing decision, whether you meant to or not. Round-robin ignores KV cache state and wastes GPU. Load-aware helps with hotspots but still misses caches. Hash-based gives you affinity but it's rigid and breaks at scale. Cache-aware routing (what llm-d does) tracks actual KV state across pods -- order-of-magnitude TTFT improvements on prefix-heavy workloads, and it runs on any Kubernetes cluster with any GPU vendor.
Honestly, if you're deploying LLMs, you've already made a routing decision -- you just might not know it. Sticking a basic load balancer in front of your vLLM replicas? That's a routing decision. Going through a managed API gateway? Also a routing decision. And each one carries trade-offs in cost, latency, vendor lock-in, and operational flexibility that compound as you scale.
So this post is my attempt to map the landscape: what's out there, where each approach wins, and how I'd choose based on the workload.
The default. Requests get spread evenly across replicas with no awareness of what each replica has cached or how busy it is.
Sends each request to the least-busy replica, judged by queue depth, GPU utilization, or pending request count.
Hashes the prompt (or a prefix of it) and consistently maps it to the same replica.
This is the interesting one. It tracks actual KV cache state across all replicas in real time and routes each request to the pod with the longest matching cached prefix, balanced against load.
The gap between approaches widens fast as prefix length grows. In illustrative comparisons at 32K tokens of shared prefix, cache-aware routing can be tens of times faster than round-robin -- the difference between waiting seconds and waiting milliseconds for the first token.
When I'm comparing routing approaches, I score them on five dimensions:
Proprietary inference platforms tend to bundle routing with their serving stack. Convenient today, sure -- but the day you need to switch hardware vendors, change models, or move clouds, you're rebuilding the routing layer from scratch.
llm-d runs on any Kubernetes cluster, any GPU vendor (NVIDIA, AMD, Intel), and any vLLM-supported model. The routing logic is decoupled from the infrastructure, which is kind of the whole point.
GPU generations keep turning over, and new accelerators keep showing up (NVIDIA B200, AMD MI325X, Intel Gaudi 3). Your routing layer shouldn't be tied to one vendor's SDK. llm-d sticks to standard Kubernetes primitives -- pods, services, endpoints -- so it works across any hardware.
If someone just needs the summary:
Vendor lock-in is the hidden cost of proprietary inference platforms. llm-d runs on any Kubernetes cluster, any GPU vendor, and any vLLM-supported model -- so your options stay open as hardware evolves.
Here's the decision tree I actually use:
I also built an interactive deck that puts the approaches side by side -- animated request flows, architecture stacks, illustrative TTFT curves, and workload fit ratings. Easier to poke at than to read about.