Slide 1 of 7
llm-d llm-d Open Source Framework

What Breaks at Scale

From proof of concept to production meltdown, and how llm-d keeps your inference stack alive at 128 GPUs and beyond.

5 Walls
Every Scaling Journey Hits
3→760
Req/s Recovered with llm-d
99.9%
Fewer Errors with llm-d
All numbers in this story are illustrative of the failure pattern, not measured benchmarks.
Use the arrow keys, or the buttons below, to navigate
Slide 2 of 7
Starting Point

It Works on My Laptop

Your proof of concept runs beautifully on a single GPU. 140ms TTFT, 80 req/s throughput, 0.2% error rate. vLLM with PagedAttention handles everything gracefully. But production has other plans.

  • Single vLLM instance, single GPU. Life is simple.
  • KV cache fits comfortably in memory
  • No inter-pod communication needed
  • Latency is fast, errors are rare
140ms
TTFT
80 req/s
Throughput
0.2%
Error Rate
Slide 3 of 7

Five Walls Every Scaling Journey Hits

You scale from 1 GPU to 128. Every step reveals a new failure mode that did not exist at smaller scale.

⚡

Network Bottleneck

Appears at 4 GPUs
High
⏳

Scheduler Contention

Appears at 8 GPUs
High
📈

KV Cache Thrashing

Appears at 16 GPUs
Critical
🌊

Thundering Herd

Appears at 32 GPUs
Critical
💥

Cascade Failure

Appears at 128 GPUs
Critical
Slide 4 of 7

Watching It All Fall Apart

An illustrative simulation: one overloaded pod triggers a chain reaction, and within about 90 seconds the whole cluster reaches zero serving capacity.

[00:00:12] NCCL WARN Net/IB : Got completion with error
[00:00:34] KV cache hit rate dropped to 8.2%
[00:01:07] 47/128 pods unresponsive, cascade in progress
[00:01:45] cluster serving capacity = 0 req/s
TTFT
140ms→5,000ms
Throughput
80 req/s→3 req/s
Errors
0.2%→60%
Slide 5 of 7

How llm-d Fixes Each Wall

Kubernetes-native building blocks — vLLM under an intelligent Inference Gateway — that remove the preconditions for cascade failure.

⇄

Inference Gateway (EPP)

The Endpoint Picker extends the Gateway API to score every pod on load, queue depth, and cache state instead of routing round-robin.

🗃

KV-Cache-Aware Routing

Sends each request to a pod that already holds matching KV-cache blocks, so shared context skips redundant prefill work.

🎯

Prefix-Aware Scheduling

Requests that share a prompt prefix land on the same pods, keeping cache hit rates high instead of thrashing across the fleet.

☰

Prefill/Decode Disaggregation

Splits compute-bound prefill from memory-bound decode onto separate workers, so long prompts stop stalling token streaming.

🗂

InferencePool CRD

A Gateway API Inference Extension resource that manages model-server pods as one pool the gateway understands, keeping endpoint churn off the control plane.

🛡

Load-Aware Routing

Steers traffic away from saturated pods before they tip over, removing the trigger that turns one slow pod into a cluster-wide cascade.

Slide 6 of 7

Before and After

An illustrative scenario at 128 GPUs, with and without llm-d. The shape of the curve is the point, not the exact numbers.

Metric Without llm-d (128 GPUs) With llm-d (128 GPUs)
TTFT (p99)5,000ms95ms
Throughput3 req/s760 req/s
Error Rate60%0.08%
Cache Hit Rate8%85%+
Scheduler Queue120 deep1.5
253x
Throughput Recovered (3 → 760 req/s)
99.9%
Fewer Errors
10x
Cache Improvement
Relative throughput per GPU (%) as the cluster grows — illustrative curves, not measured benchmarks.
Slide 7 of 7
Get Started

From 3 req/s to 760 req/s

Every stage builds on the last. llm-d does not replace vLLM. It orchestrates it into a production-grade distributed inference platform.

Inference Gateway (EPP) KV-Cache-Aware Routing InferencePool Prefill/Decode Disaggregation Prefix-Aware Scheduling
Visit llm-d.ai Read the Deep-Dive Blog
Markell Rawls
Typical Infrastructure Limits (approximate, cluster-dependent) Pods per node: ~110  |  Endpoints per service: ~5,000  |  etcd value size: ~1.5 MB
vLLM concurrent sequences per GPU: ~256  |  Scheduler throughput: ~100 pods/sec
etcd write throughput: ~10,000 writes/sec