Slide 1 of 9
llm-d Open Source Framework
Choose Your Own
Inference Stack
Plan, compare, and build production-ready LLM serving architectures with llm-d. Answer five questions and get sizing, routing, and Helm values in minutes, not weeks.
< 5 min
Time to Architecture
3
Battle-Tested Stacks
0
Vendor Lock-in
100%
Open Source
K8s
Native
Slide 2 of 9
Production LLM Serving Is Hard
Model parameters determine GPU memory, tensor parallelism, routing strategy, and whether disaggregated serving is beneficial. Five decisions shape everything.
⚖
Model Size
From 7B (single GPU) to 405B (multi-node FP8). Each jump changes your entire GPU topology, memory planning, and cost profile.
⏱
Latency SLO
Best-effort throughput vs. sub-100ms TTFT for real-time apps. Your latency target determines whether you need prefill/decode disaggregation.
👥
Multi-Tenancy
Single team, shared model with priority tiers, LoRA adapters on a shared base, or fully isolated namespaces per tenant.
💻
GPU Budget
1-2 GPUs for dev prototypes, 4-8 for single-node production, 16-32 for multi-node, 64+ for hyperscale with full disaggregation.
☁
Platform
Cloud managed K8s (EKS/GKE/AKS), self-managed, OpenShift with operators, or bare metal for ultimate control and lowest latency.
Slide 3 of 9
The Inference Stack, Layer by Layer
Every request flows through four layers, each one optimized for production inference at scale.
Gateway API
Traffic management
EPP
Smart routing
Prefill Workers
Compute-bound
Decode Workers
Memory-bound
Prometheus
Metrics + alerts
HPA
Autoscaling
EPP: The Intelligence Layer
The Endpoint Picker Plugin routes each request to the pod most likely to already hold its prefix in KV cache, dramatically cutting TTFT for multi-turn workloads. It supports load-aware, prefix-cache, LoRA-affinity, and priority scheduling strategies.
Disaggregated P/D
Separates compute-heavy prefill from memory-bandwidth-bound decode so neither blocks the other. Each phase can be independently optimized and scaled for your specific traffic pattern.
Slide 4 of 9
Which Stack Fits You?
Your model size and GPU budget determine the architecture. The decision tree maps every combination to a recommended stack.
Slide 5 of 9
Build Your Stack: 5 Questions
Answer five questions and get a complete llm-d architecture: GPU sizing, EPP routing strategy, Helm values, and deployment guidance.
Slide 6 of 9
Three Reference Architectures
From MVP to hyperscale, each stack is battle-tested and production-ready.
Starter
Startup MVP
Minimal viable inference for prototyping and early production. One or two replicas of a 7B-13B model on a single node.
Cost
$300-800/mo
GPUs
1-2x A100
TTFT
200-400ms
Throughput
5-10 req/s
vLLMEPPGateway
Enterprise
Enterprise Production
High-availability, multi-tenant deployment with monitoring and autoscaling. Priority scheduling across teams.
Cost
$8K-18K/mo
GPUs
8-16x H100
TTFT
80-200ms
Throughput
20-50 req/s
vLLMEPPGatewayKServePrometheusGrafanaHPA
Hyperscale
Hyperscale
Multi-cluster, fully disaggregated prefill/decode with LoRA routing. Built for 405B-class models.
Cost
$40K-100K+/mo
GPUs
64-128x H200
TTFT
100-300ms
Throughput
100-500 req/s
vLLMEPPGatewayKServePrometheusGrafanaHPAP/D Disagg.
Cost Efficiency
MVP 9/10
Ent. 4/10
Hyp. 1/10
Latency
MVP 5/10
Ent. 7/10
Hyp. 8/10
Throughput
MVP 4/10
Ent. 7/10
Hyp. 10/10
Simplicity
MVP 9/10
Ent. 5/10
Hyp. 2/10
Scalability
MVP 3/10
Ent. 7/10
Hyp. 10/10
Slide 7 of 9
Technology Radar
Where each llm-d component sits on the adoption curve. From proven production staples to emerging capabilities.
Adopt
vLLM, EPP, Gateway API, Prometheus: proven in production, use with confidence.
Trial
Grafana, HPA, LoRA Adapters, Prefix Caching: worth pursuing, validate in your environment.
Assess
FP8 Quantization, Tensor Parallelism, Disaggregated P/D: explore when you have specific needs.
Hold
Manual Routing, CPU Autoscaling: avoid for new deployments, better alternatives exist.
Slide 8 of 9
Migration Paths to llm-d
Wherever you are today, there is a tested path forward. Effort estimates and rollback strategies for every source framework.
Raw vLLM
3-5 days
Export your vLLM config into Helm values, deploy EPP for smart routing, add Prometheus monitoring and HPA autoscaling.
Risk: Low. Additive stack, rollback is one Helm command.
TRT-LLM + Triton
10-15 days
Convert TRT-LLM engines back to HuggingFace format. EPP routing recovers throughput through better distribution.
Risk: Medium. Engine files not portable, ensemble pipelines need reimplementation.
SGLang
5-8 days
SGLang RadixAttention maps to vLLM prefix caching. EPP replaces single-process RadixAttention with distributed cache awareness.
Risk: Low-Med. Prompt programming patterns need app-level reimplementation.
Ray Serve
5-10 days
Replace KubeRay autoscaling with K8s-native HPA. Keep Ray for training but separate inference onto llm-d.
Risk: Medium. Model chaining needs separate microservices.
Ollama
5-8 days
Move from GGUF to HuggingFace SafeTensors format. Update your app from Ollama API to OpenAI-compatible endpoint.
Risk: Medium. API format change, K8s ramp-up needed.
Any to Disaggregated
10-15 days
Complete Stage 3 first. Separate prefill and decode pools, configure EPP for disaggregated routing, tune the P/D ratio.
Risk: High. Doubles operational surface, requires 25Gbps+ networking.
Slide 9 of 9
Deploy Your Stack in Minutes
llm-d is a CNCF Sandbox project. 100% open source, zero vendor lock-in, Kubernetes-native from day one. Your inference stack, your way.
# Add the llm-d Helm repo
helm repo add llm-d https://llm-d.github.io/llm-d-deployer
# Install with your values
helm install my-model llm-d/llm-d -f values.yaml
# Verify pods are running
kubectl get pods -n llm-serving -w
vLLM
Inference Engine
EPP
Smart Routing
Gateway
Traffic Mgmt
Prometheus
Observability
HPA
Autoscaling
Markell Rawls