Slide 1 of 7

llm-d Open Source Framework

What If We Split the Work?

Disaggregated Prefill/Decode separates LLM inference into specialized GPU pools -- unlocking massive latency and throughput gains.

19.8x

Faster First Token (TTFT)

1,483

Tokens/sec at 50 Concurrent

30-60%

Throughput Improvement

Slide 2 of 7

One GPU, Two Jobs, Zero Harmony

Prefill is compute-bound. Decode is memory-bound. Running both on the same GPU means neither performs optimally.

Single GPU Pod

Compute Utilization

Prefill Phase Decode Phase

Memory Bandwidth

Resource Contention

Why This Hurts

GPU utilization yo-yos between 40-70%
TTFT spikes under concurrent load
Decode latency increases during prefill bursts
You pay for GPU time you are not fully using

Slide 3 of 7

Split the Work, Specialize the Hardware

Dedicated prefill pods process prompts. Dedicated decode pods generate tokens. KV caches transfer via RDMA in under 1ms.

Prefill Pods

Compute-optimized
Process prompts, build KV cache

→

NIXL / RDMA

GPU-to-GPU transfer
Sub-millisecond latency

→

Decode Pods

Memory-optimized
Generate tokens from KV cache

90%+

GPU Utilization

Stable

TTFT Under Load

~1ms

KV Transfer Time

Slide 4 of 7

Intelligent Routing with EPP

The Endpoint Picker (EPP) decides where each phase goes -- prefix-aware routing ensures cache hits, least-load balancing prevents hot spots.

Prefix-Aware Routing

Matches requests to pods that already hold relevant KV cache blocks, achieving up to 19.8x TTFT improvement via cache reuse.

KV-Affinity Decoding

Decode requests route to pods closest to the KV cache source, minimizing transfer latency and maximizing batch efficiency.

Least-Load Balancing

Distributes prefill work evenly across pods, preventing queue buildup and maintaining consistent TTFT at high concurrency.

Slide 5 of 7

The Numbers Tell the Story

Benchmarked on NVIDIA H200 (141 GB HBM3e) running Llama 3.3 70B FP8. Every number is from real hardware.

4.74s → 0.24s

TTFT at 32K Context (Cached)

45.5 → 1,483

Tokens/sec (1 to 50 Concurrent)

0.60s P99

Tail TTFT at 32K (Disaggregated)

Slide 6 of 7

Full System Architecture

From request ingress through the EPP router, across prefill and decode pods, with NIXL/RDMA bridging the KV cache transfer.

EPP Router

Prefill Pods (vLLM + pd-sidecar)

NIXL/RDMA Transfer

Decode Pods (vLLM + pd-sidecar)

Slide 7 of 7

Deploy It Today

Four YAML configs -- from baseline to fully disaggregated. Copy the one that matches your setup.

1

Standard Deployment

apiVersion: inference.llm-d.ai/v1alpha1
kind: LLMInferenceService
spec:
  inferenceConfig:
    replicas: 2
  routing:
    strategy: prefix-aware

2

Add Prefill Workers

spec:
  inferenceConfig:
    replicas: 2
  prefill:
    replicas: 1
  routing:
    strategy: prefix-aware

3

KV Cache Transfer

spec:
  prefill:
    replicas: 1
  kvCacheTransfer:
    connector: nixl
    transport: rdma

4

Full Disaggregation

spec:
  routing:
    strategy: disaggregated-prefill-decode
    prefillRouting:
      loadBalancing: least-load
    decodeRouting:
      loadBalancing: kv-affinity

View on GitHub Content Hub