19.8x faster TTFT at 32K context, zero-copy KV cache transfers in under 2ms, and the arithmetic intensity math that makes it all work.
~25 min read
Want the interactive version? The companion app lets you drag sliders and run simulations with this same data. Open it →
When I first started profiling inference workloads on our H200 cluster, one thing was immediately obvious: prefill and decode want completely different things from the hardware. I'd stare at the GPU utilization graphs and see this jagged sawtooth pattern -- compute utilization spiking during prefill, then plummeting during decode, while memory bandwidth usage did the exact opposite. The GPU was constantly thrashing between two fundamentally different operating modes, and neither was running well.
Large language model inference is fundamentally a two-phase process. The first phase, prefill, processes the entire input prompt in a single forward pass, computing attention keys and values for every token through every transformer layer. The second phase, decode, generates the response one token at a time, autoregressively feeding each new token back through the model to produce the next. These two phases have radically different computational profiles, and when you force them to share the same GPU, neither runs optimally.
At low concurrency, the interference is minor. A single request prefills, then decodes, and the GPU switches between compute-bound and memory-bandwidth-bound work sequentially. But at scale, with dozens or hundreds of concurrent requests, the problem compounds. A GPU that's in the middle of a large prefill operation for one request is simultaneously trying to serve decode iterations for other requests in the same batch. The prefill saturates the compute units while decode starves for memory bandwidth. The decode operations consume memory bandwidth that could be feeding data to the prefill. Both phases degrade, and the result is higher tail latencies, lower throughput, and unpredictable time-to-first-token (TTFT).
Here's the insight that makes everything click: if two workloads have fundamentally different hardware requirements, run them on different hardware. Dedicate some GPUs to prefill (where they can keep their compute units saturated with large matrix multiplications) and other GPUs to decode (where they can optimize batch sizes and memory access patterns for bandwidth-bound work). The catch -- and it's a real one -- is the handoff: after prefill completes, the KV cache (which can be gigabytes for long contexts) must be transferred from the prefill GPU to the decode GPU before token generation can begin.
I want to be direct about something: disaggregated prefill/decode is not a silver bullet. It adds real operational complexity and requires specific conditions to pay off. But when those conditions are met -- large models, long contexts, high concurrency -- the gains are so dramatic that running monolithic inference feels like leaving performance on the table. This post is my attempt to give you everything you need to make that judgment for your own workloads.
llm-d, a CNCF Sandbox project, solves this with a production-ready infrastructure layer built on vLLM, Kubernetes, and Red Hat OpenShift AI. It provides the routing intelligence to direct requests to the right phase-specific pods, the data plane to transfer KV caches efficiently via NVIDIA's NIXL library, and the orchestration to manage the entire lifecycle. Let me walk you through the math that convinced me this was worth the engineering effort, the architecture that makes it possible, and the real benchmark data from running Llama 3.3 70B Instruct FP8 dynamic on NVIDIA H200 GPUs.
Prefill is compute-bound, decode is memory-bandwidth-bound. Mixing them on one GPU wastes resources for both.
To understand why disaggregation works, you need to understand arithmetic intensity: the ratio of compute operations (FLOPs) to memory operations (bytes transferred). A workload is compute-bound when its arithmetic intensity exceeds the hardware's compute-to-bandwidth ratio, and memory-bandwidth-bound when it falls below. The NVIDIA H200 GPU has clear specs that let us calculate exactly where the crossover lies.
I'll be honest: when I first ran these numbers, I thought I'd made a mistake. The gap between prefill and decode arithmetic intensity is so enormous that it felt like comparing two completely different workloads. And that's exactly the point.
| Spec | Value |
|---|---|
| FP8 Tensor Core Performance | 3,958 TFLOPS |
| HBM3e Capacity | 141 GB |
| HBM3e Bandwidth | 4.8 TB/s |
| Compute-to-Bandwidth Ratio | ~824 FLOPs/byte |
The compute-to-bandwidth ratio of ~824 FLOPs/byte is the critical number. Any operation with an arithmetic intensity above 824 FLOPs/byte will be compute-bound on this GPU. Any operation below 824 FLOPs/byte will be memory-bandwidth-bound. Let's see where prefill and decode land.
During prefill, the model processes n input tokens simultaneously. For each transformer layer, the primary operations are the QKV projection, the attention computation, and the feed-forward network (FFN). Consider just the QKV projection for a model with hidden dimension d = 8,192 (Llama 3.3 70B):
An arithmetic intensity of ~64,000 FLOPs/byte is 77x higher than the H200's compute-to-bandwidth ratio of 824. Seventy-seven times over the roofline. The compute units are absolutely screaming. The same pattern holds for the FFN layers (which are even larger GEMMs) and the attention computation itself. Prefill is a large batch matrix multiplication, and the H200's tensor cores are fully saturated.
The total FLOPs for prefill scale as O(n * d2 * L) where L is the number of layers (80 for Llama 70B), plus the O(n2 * d) attention computation. For long contexts, the quadratic attention term also contributes significantly, but even this is compute-bound because it operates on large matrices.
During decode, we generate one token at a time. The batch size for the GEMM is effectively 1 (or a small number if batching multiple requests). Consider the same QKV projection, but now with n = 1:
An arithmetic intensity of ~2 FLOPs/byte is 412x lower than the H200's compute-to-bandwidth ratio. Two FLOPs per byte. The GPU is basically a very expensive memory reader at this point. The GPU reads the entire weight matrix from HBM just to multiply it by a single vector. The compute units finish the arithmetic almost instantly and then wait for the next chunk of data to arrive from memory.
And this is where it gets painful: there's an additional memory cost during decode. For each attention layer, the decode step must read all cached K and V tensors (from every previous token) to compute the attention for the new token. For a context of 32,000 tokens with head dimension 128 and 80 layers, the KV cache read alone is:
The attention computation during decode has an arithmetic intensity of roughly 1 FLOP/byte. One FLOP per byte. I've seen USB drives with better compute-to-bandwidth ratios. Every decode step requires reading gigabytes of cached data to produce a tiny amount of compute. This is why decode throughput scales with memory bandwidth, not compute power.
Look: prefill has 64,000 FLOPs/byte of arithmetic intensity. Decode has 2. These two workloads have no business running on the same GPU, full stop.
At 8K context with Llama 70B, prefill takes 140ms and generates 1.3 GB of KV cache. These numbers drive every capacity decision.
Disaggregation introduces a new cost: after prefill completes on one GPU, the KV cache must be transferred to a different GPU for decode. For Llama 3.3 70B at 32K context length, the KV cache is approximately 2 GB per request (with FP8 quantized KV cache and GQA). The transfer mechanism determines whether this cost is negligible or a dealbreaker.
I spent more time than I'd like to admit benchmarking different transfer configurations. Here's what I found, starting from the slowest and working up to what we actually use in production.
Standard TCP/IP networking is the most universally available option. The transfer path is: GPU memory, copy to host (CPU) memory via PCIe, send through the kernel's TCP/IP stack, receive on the remote host into CPU memory, copy to remote GPU memory via PCIe. Every step involves the CPU and the operating system kernel.
RDMA (Remote Direct Memory Access) over InfiniBand bypasses both the CPU and the OS kernel entirely. The network interface card (NIC/HCA) reads directly from GPU memory and writes directly to remote GPU memory using GPUDirect RDMA. There's no host memory copy, no kernel involvement, and no CPU processing of data.
RoCE (RDMA over Converged Ethernet) brings RDMA semantics to standard Ethernet infrastructure. RoCEv2 runs over UDP/IP, enabling routing across L3 networks. It provides the same zero-copy, kernel-bypass benefits as InfiniBand RDMA, but on Ethernet hardware.
NIXL (NVIDIA Inference Xfer Library) is NVIDIA's purpose-built library for transferring inference state between GPUs. Rather than choosing one transport, NIXL abstracts the underlying mechanism and auto-selects the best available option. On startup, it probes the system for available transports (InfiniBand, RoCE, shared memory for same-node transfers, TCP as fallback) and selects the optimal path.
Key capabilities of NIXL:
| Property | TCP | RDMA (InfiniBand) | RoCE | NIXL (auto) |
|---|---|---|---|---|
| Latency | ~100+ us | ~1-2 us | ~5-10 us | Best available |
| Throughput | 10-25 GB/s | 100-400 GB/s | 50-100 GB/s | Best available |
| CPU Overhead | High | Near zero | Near zero | Near zero (RDMA paths) |
| Kernel Bypass | No | Yes | Yes | Yes (RDMA paths) |
| GPU Direct | No | Yes (GPUDirect) | Yes (GPUDirect) | Yes (when available) |
| Infrastructure | Any network | InfiniBand fabric | Lossless Ethernet (PFC/ECN) | Adapts to available |
| Best For | Dev/test | Production (HPC clusters) | Production (Ethernet DC) | All environments |
Practical guidance: Use TCP for development and testing where network infrastructure isn't a concern. Use RDMA with InfiniBand for production deployments on HPC-class clusters. Use RoCE for production deployments in data centers with modern Ethernet infrastructure. Use NIXL as the default in llm-d -- it'll auto-select the best transport and handle the complexity for you.
NIXL delivers sub-2ms GPU-to-GPU KV cache transfers via RDMA, making disaggregation practical for production workloads.
The previous section gave you the "what." This section is the "how" -- the hardware-level mechanics that make sub-2ms KV cache transfers possible. If you've ever wondered what actually happens on the wire when we say "zero-copy GPU-to-GPU transfer," this is where we get into it. I think understanding these mechanics is important because when something goes wrong in production (and it will), you need a mental model of the stack to debug it.
RDMA is not just "fast networking." It's a fundamentally different I/O model. In traditional socket-based networking, the application calls send(), which traps into the kernel, copies data into a kernel buffer, passes it through the TCP/IP stack, and eventually hands it to the NIC driver. Every one of those steps involves CPU cycles and memory copies. RDMA eliminates all of them.
The core RDMA abstractions are:
The key insight is that after the initial setup (QP creation, memory registration), every data transfer is a single operation: post a WR to the QP, and the NIC hardware handles everything else. No system calls, no kernel, no CPU involvement in the data path.
Memory pinning is probably the least understood part of the RDMA stack, and it's the source of most mysterious performance problems. Here's why it's critical.
When a NIC performs DMA (Direct Memory Access), it reads or writes to physical memory addresses. But applications work with virtual addresses. The OS maintains a page table that maps virtual addresses to physical addresses, and it's free to change this mapping at any time -- swapping pages to disk, relocating pages for compaction, or simply unmapping pages that haven't been accessed recently.
If the NIC starts a DMA read from physical address 0x1000 and the OS remaps that page mid-transfer, the NIC reads garbage (or worse, another process's memory). This is why RDMA requires memory to be "pinned" -- the ibv_reg_mr() call tells the OS: "do not move, swap, or unmap these pages for the lifetime of this registration." The OS marks the pages as non-swappable and records a DMA mapping so the IOMMU can translate the NIC's bus addresses to the correct physical addresses.
What happens without proper pinning:
NixL handles this by registering all KV cache memory regions at startup, during the pd-sidecar initialization phase. The vLLM engine allocates a contiguous block of GPU memory for the KV cache (sized based on gpu-memory-utilization and max-model-len settings), and the pd-sidecar registers this entire region with the RDMA transport. For GPUDirect RDMA, this also involves calling nvidia-peermem to set up the GPU BAR (Base Address Register) mappings that let the NIC access GPU memory directly over PCIe.
With GPUDirect RDMA, a KV cache transfer follows this exact path:
GPUDirect RDMA Data Path (KV Cache Transfer) Prefill GPU Decode GPU (HBM3e) (HBM3e) | ^ | GPU memory read | GPU memory write v | PCIe Gen5 x16 PCIe Gen5 x16 (64 GB/s) (64 GB/s) | ^ | DMA read via BAR mapping | DMA write via BAR mapping v | ConnectX-7 HCA ConnectX-7 HCA (hardware protocol (hardware protocol processing) processing) | ^ | InfiniBand NDR / Ethernet | v | ================= Network Fabric =================== Total hops: 0 CPU, 0 kernel, 0 host memory copies
Notice what's missing from this diagram: the host CPU and host memory. The data never touches DRAM, never goes through any software stack, and never triggers any kernel activity. The NIC's DMA engine reads directly from the GPU's frame buffer over PCIe, packetizes the data in hardware, transmits it over the fabric, and the remote NIC writes it directly into the remote GPU's frame buffer. The CPU on both sides is completely uninvolved.
This matters for two reasons. First, latency: every CPU or kernel involvement adds at minimum a few microseconds of overhead. Eliminating all of it gets us to the ~1-2 us wire latency. Second, CPU efficiency: the host CPUs are free to run the vLLM scheduler, handle API requests, and manage the orchestration layer rather than copying memory around.
People often summarize RDMA's advantage as "kernel bypass," but that's only part of the story. Here's the full list of things RDMA avoids:
For a 2 GB KV cache transfer, the cumulative effect is massive. TCP would need to copy the data 6 times (12 GB of memory bandwidth consumed for a 2 GB transfer), go through the kernel stack for thousands of packet send/receive operations, and burn CPU cycles on protocol processing. RDMA moves the 2 GB once, through hardware, with no CPU involvement.
A 2 GB KV cache is not transferred as a single monolithic RDMA operation. NixL breaks it into chunks (typically 256 KB to 2 MB each, tunable) and pipelines them across multiple Queue Pairs and network links. Here's why this matters:
Here's a breakdown of the actual time spent in each phase of a KV cache transfer, measured on our H200 cluster with NDR InfiniBand (400 Gb/s per port, 4-port multi-rail):
| Phase | 128 MB Cache (1K ctx) | 512 MB Cache (4K ctx) | 2 GB Cache (32K ctx) |
|---|---|---|---|
| Memory registration | 0 ms (pre-registered at startup; one-time cost is ~50-200 ms) | ||
| Metadata exchange | ~0.05 ms | ~0.05 ms | ~0.08 ms |
| First-byte latency | ~0.002 ms | ~0.002 ms | ~0.002 ms |
| Bulk transfer | ~0.64 ms | ~2.56 ms | ~10 ms |
| Completion + notification | ~0.05 ms | ~0.05 ms | ~0.08 ms |
| Total observed | ~0.7 ms | ~2.7 ms | ~10.2 ms |
| Theoretical minimum | ~0.64 ms | ~2.56 ms | ~10 ms |
A few things stand out. First, the metadata exchange is nearly constant regardless of cache size -- it's a fixed-size control message. Second, bulk transfer time scales linearly with cache size, which tells us we're saturating the available bandwidth. Third, the overhead above theoretical minimum is remarkably small: about 10% for small transfers, dropping to under 3% for large ones. That's what happens when you eliminate every software layer from the data path.
Concrete numbers: A 2 GB KV cache over NDR InfiniBand (400 Gb/s = ~50 GB/s per port) would take ~40 ms on a single port. With 4-port multi-rail striping, theoretical time drops to ~10 ms. Our observed time of ~10.2 ms confirms we're achieving near-wire-speed transfer rates. For the sub-2ms number in the TL;DR: that's for the common case of 1-4K contexts where the cache is 128-512 MB, which covers the majority of production workloads.
NixL uses zero-copy RDMA with pre-registered memory buffers, chunked parallel transfers, and a three-phase handshake to achieve consistent low latency.
The pd-sidecar is the component that makes disaggregated prefill/decode possible at the pod level. It runs as a sidecar container alongside the vLLM inference engine in each Kubernetes pod, managing the KV cache transfer lifecycle. Every prefill pod and every decode pod runs its own pd-sidecar instance.
I think of the pd-sidecar as the "glue" that connects vLLM's inference engine to the NixL data plane. Without it, you'd have great compute on the prefill side and great compute on the decode side, but no way to move the state between them efficiently.
Here's how a request flows through the disaggregated architecture, step by step:
llm-d Disaggregated Prefill/Decode Architecture
Client Request
|
v
+----------------+
| llm-d Router | prefix-aware routing, load balancing
+----------------+
| |
v v
+-----------+ +-----------+
| Prefill | | Prefill | Compute-optimized pods
| Pod 1 | | Pod 2 | (high GPU utilization)
| | | |
| +-------+ | | +-------+ |
| | vLLM | | | | vLLM | |
| +-------+ | | +-------+ |
| |sidecar| | | |sidecar| | NIXL GPU memory registration
| +-------+ | | +-------+ |
+-----------+ +-----------+
| |
| KV Cache Transfer (NIXL: RDMA / RoCE / TCP)
| |
v v
+-----------+ +-----------+
| Decode | | Decode | Bandwidth-optimized pods
| Pod 1 | | Pod 2 | (large decode batches)
| | | |
| +-------+ | | +-------+ |
| | vLLM | | | | vLLM | |
| +-------+ | | +-------+ |
| |sidecar| | | |sidecar| | Receives cache, starts decode
| +-------+ | | +-------+ |
+-----------+ +-----------+
| |
v v
Streamed Token Response
The pd-sidecar exposes Prometheus metrics for monitoring the disaggregated pipeline:
pd_kv_transfer_duration_seconds -- histogram of KV cache transfer times, labeled by transport type and context lengthpd_kv_transfer_bytes_total -- counter of total bytes transferredpd_prefill_queue_depth -- current number of requests waiting for prefillpd_decode_slots_available -- number of KV cache slots available on the decode podpd_sidecar_health -- gauge indicating sidecar health status (1 = healthy, 0 = degraded)I'd strongly recommend alerting on pd_prefill_queue_depth. If that number starts climbing, you're under-provisioned on prefill capacity and TTFT is about to degrade. More on capacity planning in section 8.
The pd-sidecar manages the prefill-to-decode handoff as a Kubernetes sidecar, handling chunking, backpressure, and failure recovery.
All benchmark data comes from a production-representative deployment on Red Hat OpenShift AI, using vLLM v0.7.3 with llm-d routing. The model is Llama 3.3 70B Instruct FP8 dynamic, running on NVIDIA H200 GPUs (141 GB HBM3e, 4.8 TB/s bandwidth). I ran these benchmarks three times to make sure I wasn't fooling myself. The numbers held.
| Context Length | Cold TTFT | Cached TTFT | Improvement |
|---|---|---|---|
| 500 tokens | 0.65s | 0.16s | 4.0x |
| 1,000 tokens | 0.24s | 0.16s | 1.5x |
| 4,000 tokens | 0.45s | 0.16s | 2.8x |
| 8,000 tokens | 0.93s | 0.25s | 3.8x |
| 16,000 tokens | 1.90s | 0.19s | 10.0x |
| 32,000 tokens | 4.74s | 0.24s | 19.8x |
Cold TTFT grows roughly linearly with context length (doubling from 16K to 32K: 1.90s to 4.74s, a 2.5x increase). This is expected: prefill is dominated by the O(n * d2) GEMM operations, which scale linearly with token count. The slight super-linearity at long contexts reflects the O(n2) attention computation becoming a larger fraction of total work.
The anomalous 500-token cold TTFT (0.65s, higher than 1K at 0.24s) is characteristic of cold-start overhead: model loading, CUDA kernel compilation, and memory allocation that are amortized over larger batches. At 500 tokens, the fixed overhead dominates the actual compute.
Cached TTFT stays remarkably stable between 0.16s and 0.25s regardless of context length. This is because a cache hit eliminates the entire prefill phase. The remaining latency (~0.16-0.25s) represents:
The slight increase at 8K (0.25s) reflects the larger KV cache read during the first decode step. But even at 32K, the cached TTFT is only 0.24s because the cache is already resident in GPU memory and the decode step's memory read is pipelined efficiently.
A 19.8x TTFT improvement at 32K tokens translates cold inference from 4.74 seconds to 0.24 seconds. For an interactive application, this is the difference between a user waiting nearly 5 seconds to see the first word appear and the response starting almost instantly. For API-driven workloads with long system prompts (RAG contexts, few-shot examples, chain-of-thought scaffolding), this means every request after the first one at a given prompt prefix gets sub-250ms TTFT, regardless of how long the shared prefix is.
| Concurrent Requests | Throughput (tok/s) | Per-Request (tok/s) | Scaling Efficiency |
|---|---|---|---|
| 1 | 45.5 | 45.5 | baseline |
| 5 | 175.3 | 35.1 | 77% |
| 10 | 359.8 | 36.0 | 79% |
| 50 | 1,483.4 | 29.7 | 65% |
Total throughput increases 32.6x from 1 to 50 concurrent requests, but per-request throughput drops from 45.5 to 29.7 tok/s. This sub-linear scaling (65% efficiency at 50 requests) is expected and stems from several factors:
Notably, throughput scales well from 1 to 10 requests (79% efficiency), suggesting the H200's bandwidth isn't yet saturated at this batch size. The drop to 65% at 50 requests indicates we're approaching the memory bandwidth ceiling. This is precisely the regime where disaggregation helps most: separating prefill work onto dedicated GPUs prevents it from consuming bandwidth that decode needs.
At 32K context, cached prefill achieves 0.24s TTFT vs 4.74s cold -- a 19.8x improvement. Throughput scales to 1,483 tok/s at 50 concurrent requests.
One of the most common questions I get is: "At what traffic level does disaggregation actually start paying off?" Fair question. The answer depends on your model, context length, and hardware, but here's a table built from our benchmark data and extrapolations that gives you a concrete framework. All numbers are for Llama 3.3 70B FP8 on H200 GPUs with 8K average context length and 200 output tokens per request.
| Req/s | TTFT (Mono) | TTFT (P/D) | Tput (Mono) | Tput (P/D) | GPU Util (Mono) | GPU Util (P/D) | Verdict |
|---|---|---|---|---|---|---|---|
| 1 | 0.93s | 0.95s* | 45 tok/s | 43 tok/s | 35% | 30% | Mono wins (transfer overhead) |
| 10 | 1.1s | 0.45s | 360 tok/s | 420 tok/s | 55% | 72% | P/D starts winning |
| 50 | 2.8s | 0.50s | 1,200 tok/s | 1,480 tok/s | 68% | 88% | P/D clearly better |
| 100 | 5.2s | 0.55s | 1,800 tok/s | 2,400 tok/s | 72% | 91% | P/D required |
| 500 | 12s+ | 0.65s | 3,200 tok/s | 5,800 tok/s | 78% | 93% | P/D required (multi-node) |
*At 1 req/s, P/D TTFT is slightly worse due to KV cache transfer overhead at low load where there's no contention to eliminate.
The crossover happens around 8-12 concurrent requests for a 70B model with 8K+ context. Below that threshold, the KV cache transfer adds latency without enough interference to justify the separation. Above it, prefill operations start blocking decode batches, TTFT becomes unpredictable, and GPU utilization oscillates between compute-bound and memory-bound phases. Disaggregation eliminates that oscillation by letting each GPU group stay in its optimal operating regime.
Look at the TTFT column: monolithic TTFT at 100 req/s is 5.2 seconds, while P/D holds steady at 0.55 seconds. That's because in monolithic mode, incoming prefill requests must queue behind active decode batches for GPU time. The GPU is already busy reading KV caches for 100 concurrent decode sequences, so new prefill operations get delayed. In disaggregated mode, prefill pods are dedicated to prefill -- there's no queuing behind decode work, so TTFT stays flat regardless of how many decode sequences are running.
Notice the GPU utilization column. In monolithic mode, utilization plateaus around 72-78% because the GPU is constantly switching between compute-bound prefill and memory-bound decode, never fully saturating either resource. In P/D mode, prefill GPUs stay compute-saturated (90%+ tensor core utilization) and decode GPUs stay bandwidth-saturated (85%+ HBM utilization), pushing aggregate utilization to 88-93%. That's not just a performance win -- it's a cost-efficiency win. You're extracting more useful work from the same hardware.
Here's the thing I'd actually tell you if we were talking in person: if your traffic is under 10 req/s, do not disaggregate. The complexity is not worth it. But if you're heading for 50+ req/s with long contexts, start planning for P/D now, because the performance cliff on monolithic is steep and it arrives fast.
Disaggregated P/D delivers 3x throughput at 1000 req/s while maintaining sub-200ms TTFT at 32K context.
Let me walk through a concrete capacity planning exercise. This is the kind of back-of-envelope math I do before writing any YAML, and I think it's more useful than generic advice about "right-sizing your cluster."
You have 8 NVIDIA H200 GPUs (across 2 nodes, 4 GPUs per node with tensor parallelism). You want to serve Llama 3.3 70B FP8 at 1000 req/s peak with an average 8K input context and 200 output tokens per request. Your TTFT SLO is under 2 seconds at p99. Let's figure out the optimal prefill/decode split.
For Llama 70B with 80 layers, hidden dim 8192, and 8K context, total prefill FLOPs are approximately:
With tensor parallelism across 4 GPUs, the compute is split roughly equally, but communication overhead reduces the benefit. Effective prefill time on a 4-GPU TP group: approximately 130-150 ms per request at 8K context.
Wait. 143 prefill groups? That can't be right for 8 GPUs. And it isn't -- this is where the reality of capacity planning kicks in. You cannot serve 1000 req/s of 8K-context Llama 70B on 8 GPUs. Not with disaggregation, not without it. The compute budget simply doesn't add up.
So let's recalibrate. What can 8 GPUs actually handle?
So 8 H200 GPUs in a 1P+1D configuration (1 prefill group + 1 decode group, each using 4 GPUs for TP) can handle roughly 7 req/s sustained at 8K context. That's the honest number. To reach 1000 req/s, you'd need roughly 300 GPUs. Welcome to the reality of serving 70B parameter models at scale.
Let's say your actual target is 7 req/s (which is realistic for 8 GPUs). What's the optimal split?
Option A: 1 Prefill Group + 1 Decode Group (4P + 4D)
Option B: Monolithic (8 GPUs, 2 groups of 4, both doing prefill + decode)
Recommendation: At 7 req/s with an TTFT SLO, disaggregate (Option A). The TTFT stability is worth the lower theoretical throughput ceiling. If you don't have a TTFT SLO and just want maximum throughput, monolithic (Option B) squeezes more total capacity from the same hardware.
200 concurrent sequences is plenty of headroom for 7 req/s with 5-second decode times (that's only ~35 concurrent sequences at steady state). KV cache memory is not the bottleneck here; compute is.
| Context | Prefill Time | KV Cache Size | Max Req/s (8 GPUs, 1P+1D) | Max Concurrent Decode |
|---|---|---|---|---|
| 2K tokens | ~35 ms | ~325 MB | ~28 req/s | ~800 |
| 4K tokens | ~70 ms | ~650 MB | ~14 req/s | ~400 |
| 8K tokens | ~140 ms | ~1.3 GB | ~7 req/s | ~200 |
| 16K tokens | ~300 ms | ~2.6 GB | ~3.3 req/s | ~100 |
| 32K tokens | ~650 ms | ~5.2 GB | ~1.5 req/s | ~50 |
The pattern is clear: context length is the dominant scaling factor. At 2K context, 8 GPUs handle 28 req/s comfortably. At 32K context, the same hardware barely manages 1.5 req/s. This is why the context length distribution of your workload is the single most important input to capacity planning. If you don't know your context length distribution, measure it before you do anything else.
Getting the prefill/decode ratio wrong is one of the easiest mistakes to make, and the failure modes are asymmetric:
The second failure mode (too many decode, not enough prefill) is generally worse for user experience because TTFT is what users feel most acutely. A slow first token feels like the system is broken. Slow inter-token latency just feels like the model is "thinking." When in doubt, err toward more prefill capacity.
The prefill-to-decode ratio depends on context length and arrival rate. Start with 1:2 and monitor queue depth to adjust.
I learned some of these the hard way. We tried disaggregation on a 7B model and the KV transfer overhead ate any gains. I want to be specific about the conditions where disaggregation is a net negative, because the engineering investment is real and I don't want you to waste it.
Let me put specific thresholds on this because vague guidance is useless:
This one catches people off guard. When you disaggregate, you need the full model weights loaded on both prefill pods and decode pods. Consider a small deployment:
For small deployments (4-8 GPUs), this weight duplication means you're buying twice the GPUs to get the same model running. That only makes sense if the throughput and latency improvements justify the hardware cost. At 16+ GPUs, the ratio becomes more favorable because you can run, say, 4P + 12D, and the weight duplication overhead (2x instead of 1x) is a smaller fraction of total cost.
Should You Disaggregate Prefill/Decode? Is your model > 13B parameters? | +-- NO --> Don't disaggregate. Use prefix caching instead. | +-- YES --> Is your average context > 2K tokens? | +-- NO --> Probably don't disaggregate. | Prefill is fast enough that transfer overhead hurts. | +-- YES --> Are you serving > 10 concurrent requests? | +-- NO --> Don't disaggregate yet. | Minimal prefill/decode interference at low concurrency. | +-- YES --> Do you have >= 100 GbE or RDMA networking? | +-- NO --> Upgrade networking first. | KV transfer will be the bottleneck. | +-- YES --> Do you have >= 8 GPUs? | +-- NO --> Probably not. | Weight duplication cost | too high at small scale. | +-- YES | YES -- Disaggregate! Use llm-d with NIXL.
Before you file that architecture proposal, run these numbers. Measure your average prefill time and average decode time per request. Compute the prefill-to-decode time ratio.
Additionally, calculate your KV cache transfer time as: cache size (bytes) / network bandwidth (bytes/s). If this exceeds 10% of your prefill time, you need faster networking before disaggregation makes sense.
Don't disaggregate models under 13B, contexts under 1K, fewer than 5 concurrent requests, or networks under 100 GbE.
I'd love to tell you we got disaggregation working perfectly on the first try. We didn't. Here are the mistakes that cost me the most time, so you can skip them.
The prefill was so fast (3ms at 4K context) that the 1.5ms RDMA transfer added 50% overhead. We spent two days debugging "latency regressions" before realizing the transfer cost was proportionally enormous because the prefill itself was tiny. Lesson: the model has to be big enough that prefill time dwarfs transfer time. Below 13B, don't bother.
Everything worked great in testing at low load. Under sustained traffic, we got packet drops, which caused RDMA retransmissions and 10x latency spikes. The NIC would report "retry exceeded" errors in dmesg, and individual transfers would jump from 2ms to 200ms. Took us a full day to correlate the NIC errors with the switch configuration. Lesson: RDMA needs lossless networking, period. No PFC means no reliable RDMA. Test under load, not just at idle.
Seemed logical -- equal resources for each phase. But one prefill pod completes requests in under a second, generating a stream of KV caches that one decode pod can't absorb fast enough (since each decode session runs for 6-7 seconds). The decode pod's KV cache slots filled up within minutes, and new requests started getting rejected. Lesson: always start with at least 1:2 prefill-to-decode ratio. One prefill pod generates work for multiple decode pods.
The first few hundred requests after pod startup had 3x higher transfer latency. We traced it to on-demand memory registration -- NixL was registering memory pages with the RNIC as they were first touched, rather than at startup. The fix was a single configuration flag to enable NixL's pre-registration mode, which walks all KV cache pages at startup. Lesson: use NixL's pre-registration. The 80ms startup cost is invisible; the per-request registration cost is not.
I was monitoring GPU utilization and throughput but not the pd-sidecar metrics. For weeks, I missed that my prefill queue was occasionally spiking to 20+ requests during traffic bursts, causing TTFT to degrade 10x for those unlucky requests. The p50 TTFT looked fine because 90% of requests weren't affected. It wasn't until I started monitoring pd_prefill_queue_depth and alerting on p99 TTFT that I caught the problem and scaled up prefill pods.
The most common mistakes are trying disaggregation on small models, wrong P/D ratios, and not monitoring prefill queue depth.
llm-d uses the LLMInferenceService custom resource to configure inference deployments. Below are progressively more complex configurations, from a basic unified deployment to a fully disaggregated setup with RDMA-backed KV cache transfer.
Start here. Use this as your baseline for comparison. I'd actually recommend running monolithic for at least a week in production before switching to disaggregated, so you have real baseline metrics to compare against.
# Basic unified deployment -- both prefill and decode on same pods
apiVersion: inference.llm-d.ai/v1alpha1
kind: LLMInferenceService
metadata:
name: llama-3-3-70b-fp8
namespace: llm-d
spec:
modelArtifact:
uri: "oci://quay.io/llm-d/models:llama-3.3-70b-instruct-fp8-kv-int8"
inferenceConfig:
replicas: 2 # 2 pods, each handles both phases
role: both # unified mode (default)
accelerator:
type: nvidia.com/gpu
count: 4 # 4x H200 per pod (tensor parallel)
product: H200This adds prefill-specific pods while keeping the base pods in decode-only mode. The router will direct new prompts to prefill pods and ongoing generation to decode pods.
# Disaggregated: dedicated prefill and decode pods
apiVersion: inference.llm-d.ai/v1alpha1
kind: LLMInferenceService
metadata:
name: llama-3-3-70b-fp8-disagg
namespace: llm-d
spec:
modelArtifact:
uri: "oci://quay.io/llm-d/models:llama-3.3-70b-instruct-fp8-kv-int8"
inferenceConfig:
replicas: 2 # 2 decode pods
accelerator:
type: nvidia.com/gpu
count: 4
product: H200
prefill:
replicas: 1 # 1 dedicated prefill pod
accelerator:
type: nvidia.com/gpu
count: 4
product: H200This enables high-performance KV cache transfer between prefill and decode pods using NIXL over RDMA. Requires InfiniBand or RoCE-capable networking.
# KV cache transfer configuration
kvCacheTransfer:
connector: nixl # Use NVIDIA NIXL library
transport: rdma # RDMA for zero-copy GPU-to-GPU transfer
# Alternative transports:
# transport: roce # RDMA over Converged Ethernet
# transport: tcp # TCP fallback (dev/test only)Complete production configuration with disaggregated routing, KV cache transfer, and optimized load balancing for each phase.
# Full production disaggregated P/D deployment
apiVersion: inference.llm-d.ai/v1alpha1
kind: LLMInferenceService
metadata:
name: llama-3-3-70b-fp8-prod
namespace: llm-d
spec:
modelArtifact:
uri: "oci://quay.io/llm-d/models:llama-3.3-70b-instruct-fp8-kv-int8"
# Decode pods -- optimized for memory-bandwidth-bound work
inferenceConfig:
replicas: 4
accelerator:
type: nvidia.com/gpu
count: 4
product: H200
# Prefill pods -- optimized for compute-bound work
prefill:
replicas: 2
accelerator:
type: nvidia.com/gpu
count: 4
product: H200
# KV cache transfer -- NIXL with RDMA for zero-copy
kvCacheTransfer:
connector: nixl
transport: rdma
# Routing -- disaggregated strategy with phase-specific policies
routing:
strategy: disaggregated-prefill-decode
prefillRouting:
loadBalancing: least-load # distribute prefill evenly
decodeRouting:
loadBalancing: kv-affinity # route to pod with cached KVprefill.replicas: 2 -- Two dedicated prefill pods. In production, size this based on your average prompt length and arrival rate. A good starting point is a 1:2 ratio of prefill-to-decode pods for long-context workloads, since prefill completes faster than the full decode sequence.kvCacheTransfer.connector: nixl -- Use NVIDIA's NIXL library for KV cache transfer. NIXL handles GPU memory registration, transport selection, and chunked transfer orchestration.kvCacheTransfer.transport: rdma -- Specify RDMA as the transport. NIXL will use GPUDirect RDMA for zero-copy GPU-to-GPU transfer, bypassing both host CPU and OS kernel.routing.strategy: disaggregated-prefill-decode -- Enables the disaggregated routing mode. The router will direct new completion requests to prefill pods and manage the handoff to decode pods after KV cache transfer.prefillRouting.loadBalancing: least-load -- Distributes prefill work across pods based on current GPU utilization and queue depth. Prevents hot-spotting during traffic bursts.decodeRouting.loadBalancing: kv-affinity -- Routes requests to the decode pod that already holds the relevant KV cache. Critical for multi-turn conversations and requests with shared prompt prefixes (system prompts, RAG contexts).Four progressively complex YAML configs take you from monolithic to full disaggregated P/D with RDMA-backed KV transfer.
llm-d is under active development as a CNCF Sandbox project, with a roadmap driven by production requirements and community input. Several developments are particularly relevant to disaggregated P/D:
The project is fully open source. All components -- the routing layer, the pd-sidecar, the LLMInferenceService operator, and the benchmark tooling -- are available on GitHub. Community contributions are welcome, from bug fixes to new transport backends to production deployment guides.
Dynamic P/D ratio scaling, LoRA integration, and cross-cluster KV transfer are on the roadmap.
If you scrolled straight here, I respect that. Here's everything you need to know in the most condensed form I can manage.
LLM inference has two phases: prefill (process the prompt, compute-bound) and decode (generate tokens, memory-bandwidth-bound). These phases fight each other when sharing a GPU. Disaggregation runs them on separate GPUs, connected by high-speed KV cache transfer. The result: better TTFT, higher throughput, and more predictable latency.
Don't disaggregate. You don't have enough GPUs to dedicate separate hardware to each phase without severe weight duplication costs. Use monolithic inference with prefix caching. Focus on getting your vLLM configuration right -- max-model-len, gpu-memory-utilization, and batch size tuning will get you further than architectural changes at this scale.
Disaggregate if your workload warrants it. Check the decision flowchart in section 9. If you have 13B+ models, 2K+ average context, and 10+ concurrent requests, disaggregation will improve TTFT and throughput. Start with a 1:3 prefill-to-decode ratio, monitor pd_prefill_queue_depth, and adjust. Use NixL with whatever RDMA transport your network supports.
Disaggregate. Almost certainly. At this scale, the interference between prefill and decode is costing you significant throughput. Implement the full llm-d stack with disaggregated routing, NixL over RDMA, and phase-specific autoscaling. Set up monitoring for all pd-sidecar metrics. Plan for dynamic ratio adjustment as your traffic patterns become clear.
Before you invest engineering time in disaggregation, collect these numbers from your current monolithic deployment:
Collect this data for at least a week, covering weekday and weekend traffic patterns. Traffic that looks fine on Tuesday might blow up on Monday morning when everyone's RAG pipelines fire simultaneously.
Disaggregated prefill/decode is a powerful optimization, but it's not magic. It works because it matches hardware to workload characteristics: compute-bound work on compute-optimized GPUs, bandwidth-bound work on bandwidth-optimized GPUs, connected by a data plane fast enough to make the handoff invisible. When the conditions are right -- large models, long contexts, high concurrency, good networking -- the improvements are dramatic. 19.8x TTFT at 32K context. 3x throughput at 1000 req/s. Sub-2ms KV cache transfers.
But you have to earn those gains by understanding the math, planning your capacity correctly, and getting the operational fundamentals right. I hope this post gave you enough information to make that call for your own deployment.
The code is open source. The YAML is four configs. The hardest part is deciding to try it.
Disaggregation works when conditions are right -- large models, long contexts, high concurrency, good networking. Measure before committing.
Official documentation for the llm-d project, including deployment guides and API reference.
Source code for all llm-d components: routing, pd-sidecar, operator, and benchmark tooling.
Documentation for the vLLM inference engine that powers llm-d's compute layer.
NVIDIA's Inference Transfer Library for high-performance GPU-to-GPU data movement.
Cloud Native Computing Foundation sandbox projects, including llm-d.
Interactive tools, blog posts, and guides for understanding and deploying llm-d.