At 2:47 AM on a Tuesday, my phone lit up with 47 Slack notifications. The inference cluster that had been running fine for three weeks was on fire. Not a clean outage either — the slow-motion kind, where dashboards go red one panel at a time and every fix you try makes something else worse.
Twelve hours earlier, a product manager had sent a company-wide email announcing the internal AI assistant. Traffic tripled. By midnight the queue depths were climbing. By 2 AM the first pod OOMed. By 3 AM every pod in the fleet had restarted at least once and the autoscaler was thrashing so hard it looked like a seismograph.
Here's the thing: the proof of concept worked beautifully. One inference endpoint serving a 7B parameter model on one beefy GPU node. Latency was snappy, responses were coherent. I demoed it to leadership on a Thursday afternoon and by Friday morning I had approval to "take it to production." The demo ran on 4 GPUs. Production needed 48. Those two numbers live in completely different universes.
The uncomfortable truth: 5 GPU clusters are easy. 50 GPU clusters are a different universe. The failure modes aren't the same problems at a larger magnitude — they're categorically different problems that don't exist at small scale. You won't discover them in staging. You'll discover them in production, at 2 AM, when the business depends on your system staying up.
Over the past year I've watched the same failure modes wreck LLM inference deployments over and over. Not obscure edge cases — these are fundamental scaling problems every team hits the moment they move past a single-node setup. I originally wrote up five of them. But the more production incidents I dug into, and the more teams running inference at scale I talked to, two more patterns kept showing up consistently enough that they earned their own headings.
So these are the seven things that will break. And they're exactly the problems llm-d, a CNCF Sandbox project for distributed LLM inference routing, was built to solve.
Let me walk through each one — the math, the metrics, the error messages, and the war stories to prove it.
The Scale Wall
Before we get into the specific failure modes, it's worth understanding something I call the scale wall. Every distributed system hits walls at predictable orders of magnitude. Not gradual degradation. Walls. One day the system works fine; the next day, at slightly higher load, something entirely new breaks in a way you never saw coming. I've watched teams sail through single-digit GPU counts, then smash face-first into a wall the moment they crossed into double digits.
For LLM serving infrastructure, three walls matter.
The 10x Wall: Routing and Queue Management
The first wall hits somewhere between 8 and 32 GPUs. This is where naive load balancing stops working. At small scale, round-robin is fine because the variance between request costs washes out across a handful of pods. But at 10x you have enough pods that statistical clustering becomes inevitable. Two or three pods pile up expensive requests while others sit idle. Queue depths diverge. Latency percentiles split apart. Your P50 looks great while your P99 is in agony.
We hit this wall on a Wednesday afternoon with 16 GPUs serving a coding assistant. Median response was 280ms. The 99th percentile was 14 seconds. Users weren't complaining that the service was slow — they were complaining that it was "random." Sometimes fast, sometimes unusable. That randomness was the routing layer failing to account for request cost. We learned the hard way that queue-aware routing isn't an optimization at this scale. It's a requirement.
The 100x Wall: Cache Coherence and Network Fabric
The second wall hits between 64 and 128 GPUs, and it's a fundamentally different beast. At this scale you're almost certainly running tensor parallelism across nodes, which means your GPUs are talking to each other constantly. The network fabric becomes as important as the GPUs themselves. One misconfigured switch, one oversubscribed uplink, one ECMP hash collision that sends two all-reduce flows down the same path — any of these can cut your effective throughput in half.
The cache problem changes character here too. You now have dozens of pods with independent KV caches, and the probability that a returning user's conversation state lives on the pod that gets their next request drops below 10% without cache-aware routing. You start burning 30-40% of your total GPU compute on redundant prefill operations. The dashboard shows high GPU utilization and you think the system's working hard. It is. But half that work is wasted.
The 1000x Wall: Control Plane Architecture and Multi-Cluster Federation
The third wall hits at 512+ GPUs, and it's the one that forces an architectural rethink. A single Kubernetes cluster starts groaning under the weight of pod state transitions, endpoint updates, and scheduler decisions. The etcd backing store hits write limits during scaling events. Your control plane — the thing that's supposed to manage the system — becomes the bottleneck that destabilizes it.
At this scale you need multi-cluster federation, hierarchical scheduling, and a routing layer that can make decisions across cluster boundaries. The requests-per-second capacity of your control plane matters as much as the tokens-per-second capacity of your GPUs. I've seen teams with 800 GPUs and plenty of compute headroom brought to their knees by an etcd cluster that couldn't keep up with the state changes from a single scaling event.
Why the walls matter: Each wall needs a different category of solution. You can't patch your way past a wall with the same tools that got you to it. The 10x wall needs smart routing. The 100x wall needs network architecture and distributed caching. The 1000x wall needs a fundamentally different control plane. llm-d was designed with all three walls in mind — the abstractions evolve with your scale instead of forcing you to rearchitect at each threshold.
1. Queue Depth Explosion Critical
What you see: Response times go from 200ms to 2 seconds to 30 seconds. Then timeouts start. Then everything stops.
This is usually the first thing that kills you. You'll know it's happening when your Prometheus alerts start firing in sequence and your Grafana dashboard looks like this:
Look at that distribution. Pod 2 has 119 pending requests. Pod 1 has 3. That's not a capacity problem — that's a routing problem. And the math behind it is worth understanding, because it explains why the failure feels so sudden.
Start with Little's Law: L = λW, where L is the average number of requests in the system, λ is the arrival rate, and W is the average time each request spends in the system. At steady state with low load, this is fine. Your inference engine can handle maybe 10-15 concurrent requests with a service time of around 400ms. That's an arrival rate of about 25-35 requests per second. The queue is shallow and stable.
But here's where LLM inference diverges from traditional web services: service time is not constant. When a GPU is running 10 concurrent requests, each one takes 400ms. At 30, each one takes 800ms because of memory bandwidth contention and batch scheduling overhead. At 50, you're looking at 2-3 seconds per request. The queue depth isn't just growing — it's compounding. Each additional request makes every other request slower, which makes the queue grow faster, which makes every request slower still.
The math that matters: At 10 concurrent users, your queue depth is ~4. At 50 users, it's ~25. At 100 users, it's not 50 — it's ~180, because service time has tripled. At 500 users, the math breaks entirely. You're in a regime where the queue grows faster than you can drain it. This isn't a linear scaling problem. It's a non-linear feedback loop.
Round-robin load balancing makes this catastrophically worse. Picture four pods, each comfortable at 15 concurrent requests. Round-robin sends request 1 to pod A, request 2 to pod B, request 3 to pod C, request 4 to pod D. Fine when all requests are equal. They're not. A request with a 3,000-token prompt and a 500-token generation is 20x more expensive than a 50-token prompt with a 20-token response. Round-robin doesn't know that. It sends expensive requests to pods that are already struggling and leaves capacity on pods that got lucky with cheap ones.
The result looks exactly like that Prometheus output above: one pod drowning with 119 pending requests while its neighbor sits idle with 3. And by the time you notice the imbalance, the drowning pod is already past the point of recovery.
How llm-d fixes it
The Endpoint Picker (EPP) is the routing brain of llm-d. Instead of blindly distributing requests round-robin, it keeps real-time awareness of each pod's queue depth, active sequence count, and estimated compute cost. When a request arrives, the EPP routes it to the pod with the most available capacity, not the next pod in a rotation. Queue depths stabilize across the fleet, and that non-linear feedback loop never gets a chance to form. You still need enough total capacity — but the capacity you have actually gets used.
2. KV Cache Thrashing Critical
What you see: Time to first token (TTFT) is consistently high even when the system is not under heavy load. GPU utilization spikes on prefill but throughput stays low.
To get this failure, you need to know what the KV cache actually stores and why it matters so much.
During the attention computation in a transformer, each layer computes key and value vectors for every token in the sequence. For a model like Llama 2 70B with 80 layers, 64 attention heads per layer, and a head dimension of 128, the KV cache for a single token requires 2 (K+V) x 80 (layers) x 64 (heads) x 128 (dim) x 2 (bytes, FP16) = 2.6 MB per token. For a 4K context window, that's roughly 10.5 GB of KV cache per active sequence. An A100 with 80GB of VRAM can hold maybe 4-5 active long-context sequences alongside the model weights.
Now think about a multi-turn chatbot. User sends a message, model responds, user sends another message. Without cache-aware routing, that second request might land on a completely different pod — which then has to recompute the entire KV cache for the conversation history from scratch. Your vLLM logs start telling the story:
Look at the difference: cache_hit_rate=0.0% means 847ms of prefill compute. cache_hit_rate=87.5% means 94ms. That's a 9x difference on a single request. Multiply by thousands of concurrent conversations and you're burning 40-60% of your GPU cycles on redundant prefill computation.
Eviction policy matters
When GPU memory fills up, the inference engine has to evict cached sequences to make room. Simple LRU (Least Recently Used) eviction sounds reasonable until you realize it has no awareness of prefix sharing. Take ten conversations that all start with the same 500-token system prompt. LRU might evict one of those shared prefixes, forcing re-computation for multiple conversations. The log above shows exactly this: "Evicted prefix block shared by 8 active sequences." One bad eviction decision just created 8 cache misses. Prefix-aware eviction policies understand these sharing patterns and preferentially keep high-value prefixes in cache.
The distributed cache problem
Once you scale past a single node, cache coherence gets genuinely hard. If user A's conversation is cached on pod 3, but pod 3 is overloaded, do you re-route to pod 7 (which has capacity but no cache) or wait for pod 3 to free up? It's a classic tradeoff between cache locality and load balance, and the answer depends on how expensive the cache miss is versus how long the queue wait would be.
The more serious setups run hierarchical caching: L1 in GPU HBM (fastest, most limited), L2 in host memory (larger, ~10x slower to reload to GPU), and L3 in a distributed cache layer (largest, requires network transfer). Moving a KV cache from host memory to GPU takes about 5-10ms. Moving it across the network takes 20-50ms. Recomputing it from scratch takes 300-800ms. Each layer of the hierarchy is a tradeoff between access latency and capacity.
How llm-d fixes it
llm-d does prefix-aware, cache-conscious routing. The EPP tracks which pods have which conversation prefixes in their KV caches, so follow-up requests in a conversation get routed to the pod that already holds the relevant cache entries. In testing, this produces a 60-80% reduction in prefill compute for multi-turn workloads. That's the difference between needing 10 GPUs and needing 4. The routing layer continuously balances cache locality against queue depth, making the tradeoff on every request.
3. Network Topology Bottlenecks High
What you see: Inter-node communication latency spikes. Tensor parallelism throughput drops. NCCL timeouts in your logs.
You'll know this failure mode has arrived when your pod logs fill up with messages you've never seen before:
Most teams design their network for north-south traffic: requests coming in from clients, responses going back out. But LLM inference at scale generates enormous east-west traffic: communication between GPU nodes for tensor parallelism, pipeline parallelism, and distributed KV cache operations.
Here are the bandwidth numbers that matter. Model parallelism (splitting a model across multiple GPUs on different nodes) needs approximately 400 Gbps of inter-node bandwidth to avoid being communication-bound. That's the bandwidth needed for all-reduce operations during tensor-parallel inference. Meanwhile, your serving traffic — the actual requests and responses from users — might only need 25 Gbps. The east-west bandwidth requirement is 16x the north-south requirement.
Network architecture considerations
A standard spine-leaf data center network gives you non-blocking bandwidth between any two servers in the same leaf, but cross-leaf traffic shares spine uplinks. If your tensor-parallel pods land on different leaves, you're sharing those uplinks with everyone else. ECMP (Equal-Cost Multi-Path) routing tries to spread flows across available paths, but a single all-reduce operation is one flow. It takes one path, and that path might be congested.
For GPU-to-GPU communication at these bandwidths, there are two main options: InfiniBand and RoCEv2 (RDMA over Converged Ethernet v2). InfiniBand gets you lower latency (~1 microsecond), hardware-level congestion management, and adaptive routing built into the fabric. RoCEv2 runs over standard Ethernet, which means lower cost and simpler management, but it needs careful configuration of PFC (Priority Flow Control) and ECN (Explicit Congestion Notification) to avoid packet loss under heavy RDMA traffic. Packet loss in an RDMA fabric is catastrophic — a single dropped packet can stall an entire all-reduce operation.
The topology trap: If you deploy 8 pods across 8 nodes on 4 different leaf switches, your tensor-parallel all-reduce operations are crossing leaf boundaries on every step. With a 4:1 oversubscription ratio on spine uplinks (common in commodity deployments), you've just cut your effective inter-GPU bandwidth by 75%. The model spends more time communicating than computing. You'll see it as those NCCL allreduce warnings climbing from 50ms to 2,800ms. At that point, adding more GPUs makes your system slower, not faster.
How llm-d addresses it
llm-d's InferencePool abstraction and topology-aware scheduling keep pods involved in tensor-parallel inference co-located within the same network domain. The EPP routes requests to pools that have the required GPU topology available, so latency-sensitive operations never cross domains. Combined with Gateway API integration for north-south traffic, llm-d separates the serving path from the compute path at the architectural level.
4. Model Loading Thundering Herd High
What you see: Fifty new pods are Pending or ContainerCreating for 5+ minutes. Shared storage is pegged at 100% I/O. Existing pods start timing out on health checks.
This one usually hits right after the autoscaler fires. You run kubectl get pods and get a wall of misery:
Let's talk real numbers. A Llama 2 70B model in FP16 is approximately 140 GB on disk. Loading that from shared storage (NFS, Lustre, or S3) into GPU memory takes time. At a generous 2 GB/s read throughput from your storage layer, that's 70 seconds per pod just for the model weight transfer. Now imagine an autoscaler decision fires and requests 8 new pods at once.
Eight pods times 140 GB is over 1 TB of data that needs to flow from your storage layer at the same time. Your NFS server's aggregate throughput might be 20 GB/s on a good day. But here's the critical detail: the existing pods are also hitting that same NFS for checkpoint writes and health data. The storage layer becomes a bottleneck that hurts not just the new pods but the health of existing ones. Notice the RESTARTS: 3 on worker-2 above. That pod was healthy for three days until the thundering herd starved it of I/O.
Tensor parallelism complicates loading
When you're running tensor parallelism across 4 GPUs, the model is sharded — each GPU loads a different slice of each weight tensor. All 4 GPUs have to finish loading before any of them can start serving. One slow shard blocks the entire group. And because the shards are stored as separate files (or as byte ranges in a single file), you have 4x the number of random reads against storage.
Mitigation strategies
Teams have tried a few approaches: P2P model distribution, where one pod loads the model and serves it to other pods over the local network (a la BitTorrent). Shared tmpfs or local SSD caching, where the model is pre-loaded to node-local storage during node provisioning. Model sharding with lazy loading, where only the layer weights needed for the first few tokens load initially and the rest load asynchronously. Each has tradeoffs, but all of them beat the naive approach of every pod independently pulling the full model from centralized storage.
How llm-d fixes it
llm-d manages model lifecycle at the InferencePool level rather than per pod. The pool knows which nodes already have model weights cached locally, and scheduling decisions factor that in. New pods get preferentially scheduled to nodes where the model is already sitting in local storage, so there's no redundant network transfer. When a new model version rolls out, the pool coordinates a staged rollout instead of a thundering herd.
5. Scheduler Contention and Control Plane Limits Medium
What you see: kubectl get pods takes 15 seconds to return. Pod scheduling latency goes from milliseconds to minutes. Endpoints are stale; requests route to pods that no longer exist.
This is the failure mode that makes you question whether Kubernetes was the right choice (it was — you just need to understand its limits). Your first clue is usually the API server itself starting to complain:
The Kubernetes control plane wasn't designed for the pod churn rate that GPU inference at scale can generate. Here are the hard limits you'll hit:
- Scheduler throughput: The default Kubernetes scheduler processes approximately 100 pods per second. That sounds like a lot until your autoscaler decides to add 200 inference pods and scale down 150 old ones simultaneously. The scheduler queue backs up, and new pods sit in
Pendingstate for minutes. - etcd write limits: etcd, the backing store for all Kubernetes state, handles approximately 10,000 writes per second before latency degrades. Each pod state transition (Pending, Scheduled, ContainerCreating, Running, Terminating) is multiple etcd writes. With rapid pod churn across hundreds of inference pods, each reporting health status and metrics, you can saturate this limit. That
etcdserver: request timed outerror above is the canary in the coal mine. - etcd value size limits: Individual values in etcd are capped at 1.5 MB. A large
InferencePoolconfiguration with detailed routing rules, model specifications, and health thresholds for hundreds of endpoints can approach this limit. Exceeding it causes silent truncation or write failures. - Service endpoint propagation: When pods come and go, Kubernetes must update the
EndpointsorEndpointSliceobjects. These updates propagate to everykube-proxyinstance on every node, which then updates iptables or IPVS rules. In a 500-node cluster with rapid pod churn, endpoint propagation can lag by 30-60 seconds. During that window, traffic is being routed to pods that no longer exist, and your users seeconnection reset by peererrors with no explanation.
The API server bottleneck: Every kubectl command, every controller reconciliation loop, every scheduler decision, and every health check response flows through the Kubernetes API server. GPU inference pods are chattier than typical workloads because they continuously report detailed metrics (queue depth, KV cache utilization, GPU memory pressure). At scale, the API server becomes the single point of contention for the entire control plane.
How llm-d fixes it
llm-d keeps control plane pressure low by design. The EPP operates on a direct data plane between the gateway and inference pods — routing decisions don't flow through the Kubernetes API server at all. Health and capacity information moves directly between the EPP and pods via lightweight probes, not via Kubernetes resource updates. The InferencePool CRD keeps configuration compact and avoids encoding per-request state in etcd. Because the data plane is separated from the control plane, scaling to hundreds of pods doesn't linearly increase control plane load.
6. Autoscaler Oscillation High
What you see: The pod count graph looks like a sine wave. You're perpetually either 30% under capacity or 40% over capacity. Your GPU bill is unpredictable and always too high.
The Horizontal Pod Autoscaler (HPA) works well for CPU-based microservices where cold start is measured in seconds and the cost per pod is measured in cents per hour. GPU inference has neither of those properties. Here's the cycle that destroys you:
- T+0: Load increases. GPU utilization crosses the 80% threshold. HPA decides to scale from 8 pods to 12.
- T+10s: Four new pods are requested. They start downloading the model (140 GB for a 70B model).
- T+90s: The new pods are still loading. The existing 8 pods are drowning. GPU utilization is now 95%. HPA sees metrics are still above threshold and scales to 16.
- T+120s: The first batch of 4 new pods finally comes online. But HPA already requested 4 more. Now 12 pods are serving traffic that 10 could handle comfortably.
- T+180s: All 16 pods are online. Utilization drops to 35%. HPA starts scaling down.
- T+240s: HPA terminates 6 pods. Utilization climbs back up. The cycle repeats.
Meanwhile, your cost dashboard tells the real story:
This oscillation is fundamentally different from CPU autoscaling for two reasons:
- Cost per pod: A GPU inference pod costs $2-30 per hour depending on the GPU type. A CPU pod costs $0.02-0.10 per hour. Over-provisioning by 4 GPU pods for 10 minutes costs more than over-provisioning by 40 CPU pods for an hour. Oscillation in GPU land is expensive.
- Startup time: A CPU microservice cold starts in 2-5 seconds. A GPU inference pod takes 60-120 seconds to download the model, load it into GPU memory, compile CUDA kernels, and warm up. The HPA's default stabilization window (5 minutes for scale-down) wasn't designed for workloads where scale-up takes 2 minutes.
Scale-to-zero vs warm pool
Scale-to-zero sounds great for cost savings, but with a 2-minute cold start, the first user after an idle period waits 2 minutes for a response. A warm pool of minimum pods avoids this but burns money during quiet periods. The right answer depends on your traffic pattern — but the tooling to make that call intelligently (factoring in GPU cost, cold start latency, traffic predictability, and SLA requirements) just isn't built into the standard HPA.
How llm-d fixes it
llm-d integrates with the Gateway API and provides inference-aware scaling signals that go beyond simple utilization metrics. Instead of scaling on raw GPU utilization, llm-d exposes metrics like weighted queue depth, cache hit ratio, and effective throughput — numbers that account for the non-linear relationship between load and latency. The InferencePool can define warm pool minimums, graduated scaling steps, and cooldown periods tuned to model loading time. So scaling decisions actually know what it costs, in money and latency, to add or remove a GPU inference pod.
7. Cascading Failure Critical
What you see: One pod restarts, then another, then another. Within 90 seconds you go from "one pod is struggling" to "the entire fleet is down."
This is the boss fight. This is where failure modes 1 through 6 stop being individual problems and start being force multipliers for each other. Let me walk you through a realistic cascade. I've seen this play out in real production environments. Read it like a postmortem timeline, because that's exactly what it is.
pod/vllm-worker-5 OOMKilled (exit code 137). Its traffic redistributes via round-robin to the remaining 7 pods. Three of them were already at 85% capacity. They are now above 95%. PagerDuty fires.pod/vllm-worker-2 OOMKilled, pod/vllm-worker-7 OOMKilled. Five pods remain. The new pods are still loading the model. They are 50 seconds into a 90-second load process. etcd is processing a storm of pod state transitions and endpoint updates (Failure Mode 5). The EndpointSlice controller is falling behind.connection reset by peer errors. The 5 surviving pods are at 120% of safe capacity. Two more tip over: pod/vllm-worker-0 OOMKilled, pod/vllm-worker-6 OOMKilled. The on-call engineer is awake now, staring at a dashboard that is mostly red.0 endpoints available for service "vllm-inference". The service is down. Every request returns 503 Service Unavailable.pod/vllm-worker-8 OOMKilled, pod/vllm-worker-9 OOMKilled. The cycle continues. Full recovery takes another 12 minutes of manual intervention: draining the request queue, scaling down, scaling back up one pod at a time.Read that timeline again. A 30% traffic spike — not even a doubling — turned into a total outage in under two minutes. The blast radius expanded exponentially because each failure increased load on the survivors, which increased the probability of their failure, which increased load on their survivors. This isn't hypothetical. I've seen this exact sequence play out in production.
The core insight: Cascading failure isn't a separate failure mode. It's the emergent behavior when failure modes 1 through 6 aren't individually mitigated. Queue depth explosion provides the trigger. Uneven load distribution concentrates the damage. KV cache thrashing amplifies the compute cost. Slow model loading delays the cavalry. Scheduler contention prevents rapid recovery. Autoscaler oscillation means you never have the right number of pods. Remove any one of these preconditions and the cascade is survivable. Remove all of them and the cascade can't form.
How llm-d fixes it
This is where the full llm-d stack comes together as a circuit breaker architecture. The EPP's queue-aware routing prevents depth explosion. Cache-aware routing cuts wasted compute. Health-aware draining pulls pods out of rotation before they OOM. Load-aware distribution prevents hot spots. Each layer removes one precondition for the cascade. When a pod does go down (hardware failures happen), the remaining pods absorb the traffic proportionally to their available capacity, not equally. A pod at 30% utilization picks up more than a pod at 80%. New pods coming online get brought in gradually, taking a ramping share of traffic instead of the full backlog. The system bends. It doesn't break.
Real Numbers: Scale Thresholds
Not all scale is created equal. Each order-of-magnitude jump in GPU count brings new categories of infrastructure requirements. Here's a rough map of what changes and when:
| GPU Count | Networking | Routing | Storage & Scheduling |
|---|---|---|---|
| 1-8 | Single node or single rack. Standard 25 GbE sufficient. No east-west concerns. | Basic load balancer or even direct connection. Round-robin is tolerable. | Local model storage. Default Kubernetes scheduler. Standard HPA. |
| 8-32 | Multi-node within one leaf switch. 100 GbE recommended for tensor parallelism. | Intelligent routing becomes critical. Queue-aware and cache-aware routing yield measurable gains. | Shared model storage (NFS or object store). Need model caching strategy. Custom HPA metrics. |
| 32-128 | Spine-leaf topology required. Cross-leaf traffic needs careful management. Consider RoCEv2 for tensor parallelism. | Distributed KV cache becomes necessary. Cache-aware routing with prefix sharing. Health-aware draining. | P2P model distribution. Topology-aware scheduling (GPU locality, NUMA awareness). Multi-queue HPA with inference-specific metrics. |
| 128-512 | RDMA fabric (InfiniBand or RoCEv2 with PFC/ECN). Dedicated GPU fabric separated from management network. | Hierarchical routing: L1 intra-pool, L2 cross-pool. Circuit breaker patterns mandatory. | Custom scheduler or scheduler extender for GPU-aware placement. Distributed etcd or federated control plane. Warm pool management. |
| 512+ | Dedicated network fabric with adaptive routing. Fat-tree topology with full bisection bandwidth. Separate fabrics for compute and serving. | Multi-cluster federation with global routing policy. Geo-aware request routing. Cross-cluster KV cache migration. | Hierarchical scheduling (cluster, node pool, node, GPU). Multi-cluster model lifecycle management. Custom autoscaler with cost-aware scaling. |
Where llm-d fits: llm-d is designed to carry you from the 8-GPU threshold through the 512+ GPU regime without rearchitecting your serving stack at each stage. The EPP, InferencePool, and Gateway API integration are the abstractions that scale across these thresholds. You add infrastructure underneath, but the routing and management layer stays consistent.
Survival Checklist: Pre-Flight for Production Scale
The scale thresholds table tells you what changes. This checklist tells you what to actually do about it before you get paged at 2 AM. Treat it as a pre-flight checklist, not a feature list. If you can't check every box in a tier, you're not ready to operate at that scale.
Before you scale past 8 GPUs
- Replace round-robin load balancing with queue-depth-aware routing (or deploy llm-d's EPP)
- Instrument
time_to_first_token_p99,request_queue_depth, andgpu_memory_utilizationper pod, not just as fleet averages - Set up alerts on per-pod queue depth divergence: if any single pod's queue is 5x the fleet average, your routing is broken
- Configure model caching on node-local storage (SSD or
tmpfs) so scale-up does not hit shared storage - Set HPA cooldown periods to at least 2x your model loading time (e.g., 3 minutes if model load takes 90 seconds)
- Test your system under 2x expected peak load. If it degrades gracefully, you are ready. If it cascades, you are not.
Before you scale past 32 GPUs
- Deploy cache-aware routing that tracks KV cache state per pod and routes multi-turn conversations to cache-warm pods
- Implement prefix-aware cache eviction; verify that shared system prompts are not being evicted under pressure
- Ensure all tensor-parallel pod groups are co-located within the same leaf switch (run
kubectl get nodes -o wideand map to your network topology) - Upgrade to 100 GbE (minimum) for inter-node GPU communication; monitor NCCL allreduce latency and alert if it exceeds 100ms
- Implement health-aware draining: pods at >90% GPU memory should stop receiving new requests before they OOM
- Set up P2P or BitTorrent-style model distribution so that scaling from 8 to 16 pods does not saturate your storage layer
- Monitor etcd write latency. If it exceeds 100ms during scaling events, you are approaching control plane limits.
- Run a chaos test: kill 2 of your 8 pods simultaneously and verify the fleet absorbs the traffic without cascading
Before you scale past 128 GPUs
- Deploy a dedicated RDMA fabric (InfiniBand or RoCEv2 with PFC/ECN configured and tested) for GPU-to-GPU communication, separate from your management network
- Implement hierarchical routing with circuit breakers; no single pod failure should be able to trigger a cascade
- Move to a federated or distributed etcd topology; a single etcd cluster will not handle the write volume from 100+ inference pods churning state
- Deploy llm-d's full stack (EPP + InferencePool + Gateway API) or equivalent. You are past the point where custom scripts and tuned HPAs are sufficient
- Implement warm pool management with cost-aware autoscaling; you should know your cost-per-request at every scale tier
- Build a topology-aware scheduler (or scheduler extender) that understands GPU locality, NUMA domains, and network proximity
- Establish an incident runbook for cascading failure with specific manual overrides: how to drain the request queue, how to scale down safely, how to bring pods online one at a time
- Run a full cascade simulation: spike traffic 50% above peak, kill 3 pods simultaneously, and trigger a model rollout, all at once. Your system should survive.
How llm-d Addresses It All
Each failure mode above has a specific architectural answer in llm-d. Here's how the components map to the problems:
The Endpoint Picker (EPP) is the routing engine at the heart of llm-d. It's not a load balancer in the traditional sense — it's a real-time decision engine that considers queue depth, KV cache state, GPU memory pressure, and pod health before routing each request. This single component directly addresses failure modes 1 (queue depth), 2 (KV cache), and 7 (cascading failure), because every routing decision is load-aware, cache-aware, and health-aware at the same time.
The InferencePool is a Kubernetes Custom Resource that groups inference pods into logical pools with shared configuration, scaling policies, and routing rules. Pools can be organized by model, by GPU topology, or by tenant. This abstraction addresses failure modes 3 (network topology, by enforcing co-location), 4 (thundering herd, by managing model lifecycle at the pool level), and 6 (autoscaler oscillation, by defining inference-aware scaling parameters).
Gateway API integration connects llm-d to the Kubernetes Gateway API, providing a standard interface for north-south traffic management. This separation of concerns means that serving traffic and inter-GPU traffic are managed independently, addressing failure mode 3 (network bottlenecks) at the architectural level. It also provides the extensibility point for custom routing policies, rate limiting, and traffic shaping.
Health-aware routing with graduated draining is the mechanism that prevents failure mode 7 (cascading failure). Instead of a binary healthy/unhealthy check, the EPP implements a gradient: pods under moderate pressure get fewer requests, pods near their limits get none, and pods that are failing are drained gracefully. This prevents the "one pod fails and the survivors immediately overload" dynamic that drives cascading failure.
The design philosophy behind all of these components is the same: the infrastructure layer between users and models has to understand what LLM inference actually needs. Standard Kubernetes primitives (Services, Ingresses, HPAs) were designed for stateless HTTP microservices. LLM inference is stateful (KV cache), asymmetric (requests have wildly different costs), memory-intensive, and latency-sensitive in ways traditional web services just aren't. llm-d bridges that gap.
From the Trenches: 5 Failure Case Studies
Theory is useful. Postmortems are better. Here are five real-world failure scenarios drawn from production deployments. Names and identifying details are changed, but the metrics, the timelines, and the lessons are real. Each one left scars.
Case Study 1: The Black Friday KV Cache Meltdown
At 3 AM on Black Friday, Meridian's AI shopping assistant started giving customers product descriptions from the wrong categories. Handbags described as power tools. Running shoes with nutritional information. The engineering team initially thought the model had corrupted weights, but the truth was worse.
Traffic had tripled over two hours. The KV cache hit 98% utilization across all pods, triggering aggressive eviction. But the eviction policy was pure LRU with no prefix awareness. Meridian's system prompt included a 1,200-token product catalog context that was shared across every conversation. When that shared prefix was evicted, every active session lost its cache state simultaneously. The prefill storm consumed 78% of available GPU compute. Response latency jumped from 340ms to 9.2 seconds. The autoscaler saw high GPU utilization and added 12 more pods, which triggered a thundering herd against their NFS server. The NFS server hit 100% I/O utilization, and existing pods started failing health checks because they couldn't write cache checkpoints.
Total impact: 47 minutes of degraded service. 23% of sessions served incorrect context. Estimated revenue impact: $2.1M in abandoned carts during peak shopping hours.
Resolution: Meridian deployed prefix-aware cache eviction that pinned the shared product catalog prefix in cache. They implemented cache-aware routing via llm-d's EPP so returning customers hit the pod that held their session state. They moved model weights to node-local NVMe storage and eliminated the NFS dependency for model loading. Post-fix, the same traffic pattern produced zero cache-related incidents across Cyber Monday and the following holiday season.
Case Study 2: The Chatbot That Ate Its Own Queue
Lakeshore deployed an internal compliance chatbot for their loan officers. It worked beautifully in the pilot with 50 users. Then they rolled it out company-wide to 3,000 loan officers on a Monday morning. By 9:15 AM, the chatbot was unusable.
The root cause was deceptively simple. When the chatbot returned a slow response, the frontend UI showed a spinning indicator. Loan officers, trained to click again when things stall, would close the tab and open a new one. The frontend didn't cancel the original request. So every slow response generated a retry, which added load, which made responses slower, which generated more retries. Within 20 minutes, the system was processing 4x the actual user demand because 75% of in-flight requests were orphaned retries that nobody was waiting for.
The queue depth on two pods hit 340 pending requests. The P99 latency was 45 seconds. The system was technically "up" since pods weren't crashing, but it was functionally dead. No loan officer was willing to wait 45 seconds for a compliance answer.
Total impact: 3 hours of unusable service on launch day. 2,200 support tickets. The project was nearly canceled by executive leadership.
Resolution: Three changes saved the project. First, they implemented request deadlines: any request older than 15 seconds was automatically shed from the queue, freeing capacity for fresh requests. Second, the frontend was updated to cancel in-flight requests when a user navigated away. Third, they deployed llm-d's EPP with queue-depth-aware routing, which prevented the 340-request pileup on individual pods. Post-fix, the system handled 3,000 concurrent users with a P99 of 2.8 seconds.
Case Study 3: The Silent Network Partition
The dashboard turned red in a pattern I'd never seen before. Half the pods were reporting normal latency. The other half were timing out. But the split wasn't random. It was perfectly correlated with rack placement.
Stratos was running 8-way tensor parallelism across nodes in different racks. A firmware update on one of the spine switches had silently changed the ECN marking threshold from 100KB to 10MB. RoCEv2 traffic between racks 2 and 4 was seeing 0.3% packet loss, which doesn't sound like much until you realize that a single dropped packet in an all-reduce operation stalls every GPU in the tensor-parallel group for the full NCCL timeout (30 seconds by default). The pods weren't crashing. They were spending 90% of their time waiting for retransmissions the network should have prevented.
Effective throughput on the affected pods dropped from 1,200 tokens per second to 80 tokens per second. The monitoring system showed GPU utilization at 95%, which looked healthy. But the GPUs weren't computing — they were stalled, waiting on network operations that were being silently corrupted.
Total impact: 6 hours of 15x degraded throughput on half the fleet. The switch firmware issue wasn't detected for 4 hours because GPU utilization metrics looked normal.
Resolution: Stratos added NCCL allreduce latency as a first-class monitoring metric with alerts at the 100ms threshold. They deployed llm-d's topology-aware scheduling to ensure tensor-parallel groups stayed within a single rack, eliminating cross-rack all-reduce operations entirely. They also implemented a pre-deployment network validation check that runs synthetic RDMA traffic across all rack boundaries and alerts on any packet loss above 0.01%. The firmware issue was rolled back within an hour of detection.
Case Study 4: The Autoscaler Death Spiral
NovaBridge offered LLM inference as a managed service. Their autoscaler was configured to scale on GPU utilization with a 70% target. On a Thursday afternoon, one of their larger tenants kicked off a batch processing job that sent 10,000 requests in 5 minutes. The autoscaler responded correctly, scaling from 12 to 24 pods. But then the batch job finished.
GPU utilization dropped to 25%. The autoscaler began scaling down. It terminated 8 pods. But those 8 pods had warm KV caches for the other tenants' ongoing conversations. Suddenly, cache hit rates across the platform dropped from 72% to 18%. Prefill compute spiked. GPU utilization climbed back to 85%. The autoscaler scaled back up. Eight new pods came online with cold caches. The warm cache state was permanently lost.
This cycle repeated four times in 90 minutes. Each cycle, more conversation state was destroyed by scale-down events, and the subsequent scale-up never restored it. The platform's effective throughput during the oscillation was 40% lower than steady-state, despite having the same average number of GPUs.
Total impact: 90 minutes of degraded performance across all tenants. GPU costs for the day were 2.3x the baseline due to oscillation. Three enterprise customers filed SLA violation complaints.
Resolution: NovaBridge made three architectural changes. They switched their autoscaler signal from raw GPU utilization to llm-d's weighted queue depth metric, which accounts for cache state and does not spike on prefill-heavy workloads. They implemented a 10-minute cooldown window on scale-down (up from the default 5 minutes) and added a rule that scale-down events couldn't remove more than 25% of the fleet at once. Finally, they deployed cache-aware draining: before a pod was terminated, its active conversation state was migrated to a surviving pod's host memory cache, preserving cache warmth across scale events.
Case Study 5: The Model Update That Took Down Production
At 2 AM on a Tuesday (it's always 2 AM on a Tuesday), Clearpath's CI/CD pipeline triggered a model update. The new model checkpoint was 3% larger due to additional fine-tuning layers. The rolling update strategy was simple: terminate one pod, start a new one with the new model, wait for it to be healthy, move to the next pod.
The problem: the new model's memory footprint, while only 3% larger in weights, needed 11% more KV cache memory because the fine-tuning had added attention heads. The first 6 pods updated successfully. Pod 7 loaded the new model, started serving, and immediately hit 96% GPU memory. It began evicting KV cache aggressively. Pods 1 through 6 (already on the new model) were under the same pressure but hadn't shown symptoms yet because traffic was light.
Then morning traffic arrived at 7 AM. All 8 updated pods hit their memory ceiling at once. KV cache eviction rates spiked. Prefill compute consumed 60% of GPU cycles. The remaining 4 pods on the old model were healthy but running a different model version, so the routing layer couldn't mix traffic between old and new model pods. Effectively, only 8 pods were serving the new model, and those 8 pods needed 10 pods' worth of GPU memory.
Total impact: 4 hours of degraded performance during peak clinical hours. Time to first token went from 180ms to 6.4 seconds. The compliance team flagged the incident because clinical decision support latency exceeded their SLA of 3 seconds.
Resolution: Clearpath implemented a three-stage model deployment process. Stage one: deploy the new model to a canary pool of 2 pods and run a synthetic load test that measures actual GPU memory consumption under realistic KV cache pressure, not just model weight size. Stage two: if the canary passes, roll out to 50% of the fleet with automatic rollback if P99 latency exceeds 2x the baseline. Stage three: complete the rollout with a 30-minute bake period before removing the old model pods. They also added a pre-deployment memory audit that calculates model_weights + max_kv_cache_at_target_concurrency and rejects any model that exceeds 88% of available VRAM.
Load Testing Strategies for LLM Inference
Every team I've worked with has the same regret after their first production outage: "we should have load tested more aggressively." The problem is that most teams load test their LLM deployments the same way they load test web APIs — throw a fixed rate of identical requests at the endpoint and check if latency stays below a threshold. That approach misses every failure mode in this article.
LLM inference load testing needs a fundamentally different approach because the workload is fundamentally different. Here's what actually works.
The right tools
k6 (from Grafana Labs) is my go-to for LLM load testing because of its scriptable scenarios and built-in support for streaming responses. You can write a k6 script that simulates multi-turn conversations, varying prompt lengths, and realistic think time between messages. The key feature is scenario-based testing: you can define "20% of users send long documents, 60% send short questions, 20% are multi-turn conversations" and run all three simultaneously.
Locust is better when you need to simulate messy user behavior. Its Python-based task definitions make it easy to build conversation flows where each user's next request depends on the model's previous response. I've used Locust to simulate the "retry storm" pattern from Case Study 2 by programming users that re-send requests when latency exceeds a threshold.
vegeta is the right choice for raw throughput testing. When you need to answer "what's the absolute maximum requests per second this cluster can handle before it falls over," vegeta's constant-rate attack mode will find that number. It's less useful for realistic workload simulation but invaluable for finding hard limits.
Testing patterns that actually find bugs
The ramp-to-failure test. Start at 10% of expected peak load. Increase by 10% every 5 minutes. Don't stop when latency starts degrading. Keep going until something breaks. The goal isn't to pass a test — it's to find your breaking point before production finds it for you. Record the exact load level where P99 latency goes non-linear. That's your true capacity ceiling, and it's almost always 30-40% below what your napkin math predicted.
The burst test. Run steady-state load, then inject a 3x traffic spike for 60 seconds. Watch what happens during the spike, but more importantly, watch what happens after the spike ends. A healthy system returns to baseline within 30 seconds. A system with queue depth issues takes 5-10 minutes to drain its backlog. A system with cascading failure tendencies never recovers without manual intervention.
The chaos injection test. Under moderate load, kill 25% of your pods simultaneously. This simulates the "first blood" moment from the cascading failure timeline. Your remaining pods should absorb the traffic without any of them exceeding 90% GPU memory. If they do, your fleet is too close to capacity and your safety margin isn't enough.
The cache pollution test. Send 10,000 unique, non-repeating prompts to measure performance without any cache benefits. Then switch to a realistic workload with multi-turn conversations and shared system prompts. The delta between the two tells you how dependent your performance is on cache hit rates — and therefore how vulnerable you are to cache thrashing under pressure.
The load testing rule of thumb: If your load test doesn't make you uncomfortable, it's not aggressive enough. The test that makes your on-call engineer nervous at 2 PM is the test that saves them from a real incident at 2 AM. Test at 2x your expected peak, test with chaos, test with realistic workload distributions. The failure modes in this article aren't theoretical. They're waiting for the right conditions, and load testing is how you create those conditions on your terms.
Graceful degradation: when you can't scale fast enough
Sometimes the traffic arrives faster than you can add capacity. The autoscaler fires, but model loading takes 90 seconds, and the cascade is already underway. This is where graceful degradation separates the systems that survive from the ones that end up in a postmortem. The question isn't "can we handle this load?" It's "what do we do when we can't?"
We learned this the hard way during a product launch. Traffic spiked 5x in under three minutes. Our autoscaler responded correctly, but the new pods wouldn't be ready for 90 seconds. So we had 90 seconds of 5x load on infrastructure sized for 1x. Without a degradation strategy, the whole system would have collapsed. Here's what we built afterward.
Request Shedding
When queue depths exceed a configurable threshold, new requests get rejected immediately with a 429 Too Many Requests instead of being queued. Sounds brutal, but the alternative is worse. Queuing a request you can't serve for 30 seconds doesn't help the user. It eats GPU memory for the pending KV cache allocation, makes every other queued request slower, and returns a timeout the client treats as an error anyway. A fast rejection lets the client retry intelligently, show a "service is busy" message, or fall back to an alternative. The key is setting the shedding threshold correctly. Too aggressive and you reject requests you could have served. Too permissive and you're back to queue depth explosion.
Quality Tiering: Smaller Model Fallback
This is the most effective degradation strategy I've seen in production. When load exceeds primary model capacity, overflow traffic gets routed to a smaller, faster model. A 70B model might produce the best responses, but a 7B model running on the same GPU can handle 8x the throughput. The responses are less nuanced, but they're fast and functional. Users get a slightly lower quality answer in 400ms instead of a timeout in 30 seconds — or worse, no answer at all.
You do have to maintain a warm pool of fallback model pods, which costs money during normal operations. But the alternative is a total outage during traffic spikes, which costs far more. In practice, the fallback model handles 5-10% of total traffic during peak hours, and the warm pool runs roughly 15% of the primary fleet's GPU cost.
Priority Queues
Not all requests are equal. A paying enterprise customer's request is worth more than a free-tier exploration query. A clinical decision support query in a healthcare app is more urgent than a "summarize this article" request. Priority queues let you guarantee latency for high-priority traffic even when the system is under pressure, by shedding lower-priority requests first.
The implementation is straightforward: tag each request with a priority level at the gateway, and configure the EPP to route high-priority requests to reserved capacity. During normal operations, the reserved capacity handles overflow from the general pool. During overload, the general pool sheds requests while the reserved pool keeps serving priority traffic at target latency.
Circuit Breakers
A circuit breaker watches error rates and latency on downstream services. When failures exceed a threshold, the circuit "opens" and stops sending traffic to the failing component. This is what prevents the cascading failure pattern from section 7. Without circuit breakers, a single unhealthy pod keeps receiving traffic, fails to respond, and the retry storms from upstream services amplify the failure. With circuit breakers, the unhealthy pod gets pulled from rotation within seconds and traffic redistributes to healthy pods proportionally.
The parameters that matter are the failure threshold (what error rate trips the circuit), the open duration (how long to stop sending traffic), and the half-open behavior (how to test whether the downstream service has recovered). For LLM inference, I set the failure threshold at 3 consecutive timeouts or a 50% error rate over a 10-second window. The open duration is 30 seconds — long enough for a struggling pod to drain its queue, short enough to restore capacity quickly if the issue was transient.
The degradation hierarchy: When load exceeds capacity, apply these strategies in order. First, activate priority queues to protect your most important traffic. Second, begin shedding low-priority requests. Third, route overflow to fallback models. Fourth, if even the fallback models are saturated, open circuit breakers on the most stressed pods and let them recover. Each layer buys you time. Time for the autoscaler to add capacity. Time for the traffic spike to subside. Time for the on-call engineer to intervene. The system degrades gracefully instead of collapsing catastrophically.
Capacity Planning Formulas
The war stories above share a theme: teams that got surprised by the math. Capacity planning for LLM inference isn't guesswork. These are the formulas I use to size deployments, and honestly they've saved me from more incidents than any monitoring dashboard.
GPU count estimation
Start here. This formula gives you the minimum number of GPUs needed to sustain your target request rate:
The critical variable is target_utilization. Never set this above 0.7. At 80% utilization you have no headroom for traffic spikes. At 90% you're one bad minute away from queue depth explosion. I use 0.6 for production systems that need to absorb unexpected bursts, and 0.7 for systems with predictable, steady-state traffic. For example: 100 peak RPS, 500 average tokens per request, 2,000 tokens per second per GPU (typical for a 70B model on an H100), target utilization of 0.65. That gives you (100 * 500) / (2000 * 0.65) = 38.5, rounded up to 40 GPUs. Add 20% for the headroom your load tests will prove you need, and you land at 48.
Memory sizing
GPU memory is the resource that kills you silently. Model weights are the obvious consumer, but KV cache is what actually determines how many concurrent requests you can serve:
For a 70B parameter model in FP16 (2 bytes per parameter), the model weights consume 140 GB. Spread across 4 GPUs with tensor parallelism, that's 35 GB per GPU. On an 80 GB A100, you have 45 GB left for KV cache. At 2.6 MB per token and a 4K context window, each active sequence needs roughly 10.5 GB of KV cache. So each GPU can hold approximately 4 concurrent long-context sequences. With 4 GPUs, your maximum concurrency for long-context requests is 16. If your queue depth exceeds 16, you're evicting caches. Plan accordingly.
Network bandwidth
For tensor-parallel inference across nodes, the all-reduce communication bandwidth determines whether your GPUs are computing or waiting:
That 164 Gbps is the minimum for the all-reduce to not be the bottleneck. In practice you want 2x headroom, because network traffic is bursty and other operations (KV cache transfers, health checks, metrics) share the fabric. This is why 100 GbE is the floor for multi-node tensor parallelism, and why serious deployments run 400 Gbps InfiniBand.
The planning rule: Run these formulas, add 20% headroom, then validate with the load testing strategies above. The formulas give you a starting point. The load tests tell you whether your starting point was right. If your load test breaks the system at 70% of the capacity your formulas predicted, dig into the gap — it's usually the KV cache memory constraint or network bandwidth that the formulas underestimated. Adjust and re-test until the math matches reality.
Key Takeaways
- Scale introduces new failure categories. The problems at 50 GPUs aren't bigger versions of the problems at 5 GPUs — they're fundamentally different failure modes you won't see in staging or small-scale testing.
- Queue depth explosion is non-linear. LLM service time increases with load, creating a compounding feedback loop where each additional request makes every other request slower. Round-robin load balancing accelerates the collapse.
- KV cache misses are the silent throughput killer. A cache miss can cost 9x more compute than a cache hit. Without cache-aware routing, multi-turn workloads burn 40-60% of GPU cycles on redundant prefill operations.
- Network topology matters as much as GPU count. Cross-leaf tensor parallelism with oversubscribed spine uplinks can cut effective bandwidth by 75%, making your system slower as you add GPUs.
- The Kubernetes control plane has hard limits. Scheduler throughput, etcd write capacity, and endpoint propagation lag all become bottlenecks at scale. Separate your data plane from your control plane.
- Autoscaler oscillation wastes 50%+ of GPU spend. Standard HPA tuning parameters weren't designed for workloads with 90-second cold starts and $30/hour pod costs.
- Cascading failure isn't a separate problem. It's the emergent behavior when the other six failure modes aren't individually mitigated. Address each precondition and the cascade can't form.
- Graceful degradation separates production from demo. Request shedding, quality tiering, priority queues, and circuit breakers each buy time for recovery. Without them, any traffic spike becomes a total outage.
- llm-d was built for these exact problems. Its Endpoint Picker, InferencePool, and Gateway API integration give you cache-aware routing, inference-native autoscaling, and circuit breaker patterns designed around how LLM serving actually behaves at scale.
Scale Readiness Checklist
Use this checklist to assess your deployment's readiness before scaling. Hover over items to check them off as you verify each capability.
Routing and Load Balancing
- Queue-depth-aware routing replaces round-robin
- KV cache-aware routing for multi-turn workloads
- Health-aware draining removes pods before OOM
- Request shedding configured with appropriate thresholds
- Priority queue support for critical traffic
Caching and Memory
- Prefix-aware cache eviction protects shared prompts
- Per-pod KV cache utilization monitoring (not fleet averages)
- GPU memory budget accounts for model weights + max KV cache at target concurrency
- Hierarchical caching strategy (HBM, host memory, distributed cache)
Networking
- Tensor-parallel pod groups co-located within same network leaf
- 100 GbE minimum for inter-node GPU communication (400G InfiniBand for 128+ GPUs)
- NCCL allreduce latency monitored with alerts at 100ms threshold
- East-west GPU fabric separated from north-south serving traffic
Model Lifecycle
- Model weights pre-cached on node-local storage (NVMe/SSD)
- P2P or staged model distribution prevents storage stampede on scale-up
- Canary deployment pipeline validates memory footprint under realistic load
- Rollback automation triggers on P99 latency regression
Control Plane and Autoscaling
- etcd write latency monitored with alerts at 100ms
- Autoscaler uses inference-aware metrics (weighted queue depth, cache hit ratio)
- Scale-down cooldown set to at least 2x model loading time
- Scale-down events limited to 25% of fleet per step
- Data plane (routing decisions) decoupled from Kubernetes API server
Resilience and Testing
- Circuit breakers configured on all downstream inference pods
- Fallback model warm pool maintained for overflow traffic
- Load tested at 2x expected peak with realistic workload distribution
- Chaos tested: 25% pod kill under load without cascade
- Cascade simulation passed: traffic spike + pod kills + model rollout simultaneously
- Incident runbook documented with manual override procedures
The Gap Between Demo and Production
Every one of these seven failure modes has the same root cause: a fundamental mismatch between what LLM inference needs and what standard infrastructure provides. You can't patch that gap with clever Kubernetes configs or custom HPA rules. You need a routing and management layer that speaks the language of inference — one that understands queue depth, cache locality, memory pressure, model loading time, network topology, and the non-linear relationship between load and latency.
That's what llm-d is. And it's being built as an open project under the CNCF Sandbox because these problems are universal to anyone running LLM inference at scale, whatever model they're serving and whatever cloud they're on.
If any of these failure modes sound familiar, you're not alone. If you're still in the POC phase, run through the survival checklist above before you scale past that first rack of GPUs. And if you're already in production and have the scars to prove it — llm-d was built by people with those same scars.
The problems don't announce themselves politely. They arrive all at once, at 2 AM, on a Tuesday. The question is whether your infrastructure knows what to do when they show up.
See these failure modes in action
I built a companion interactive app that lets you simulate each failure mode and watch llm-d's routing fix them in real time.
Further Reading
- llm-d Project Homepage Official site for the llm-d CNCF Sandbox project for distributed LLM inference routing.
- llm-d GitHub Repository Source code, architecture docs, and contribution guidelines.
- vLLM Documentation Documentation for the vLLM inference engine, including KV cache management and tensor parallelism.
- Kubernetes Gateway API The standard API that llm-d integrates with for north-south traffic management.
- NVIDIA NCCL Documentation Reference for GPU collective communication, allreduce operations, and network troubleshooting.
- Kubernetes HPA Documentation Official docs for the Horizontal Pod Autoscaler, including stabilization windows and scaling policies.
- llm-d on CNCF The CNCF Sandbox listing for llm-d with project status and community links.