Stop Overthinking Your Inference Stack. Here's a Decision Tree That Works.

1. Introduction: The Problem with "Just Deploy It" #

I've watched this same movie so many times now. The proof-of-concept works. A single vLLM instance on a rented GPU handles the demo traffic. Everyone's happy. Then the questions start: what happens when traffic doubles next Tuesday? How does the fraud team get their own fine-tuned adapter without doubling GPU spend? Why does time to first token spike when the support team's shift starts? Why is GPU utilization sitting at 30% while users complain about latency?

The gap between "vLLM can serve this model" and "we have a production inference platform" is huge. And that gap is where teams lose months. Not because the tech is hard — because decision paralysis is hard. Too many components, too many framework comparison posts, too many "it depends" answers that never actually help you ship.

So, being direct: vLLM is an inference engine, and a really good one. But an engine is not a car. You still need routing that knows where your KV cache lives. Autoscaling that reacts to inference metrics, not generic CPU utilization. Monitoring that tracks time to first token and tokens per second, not just HTTP 200 rates. And some multi-tenancy story that isn't "spin up a separate cluster for every team that wants a LoRA adapter."

That's the gap llm-d fills. It's a CNCF Sandbox project that assembles the whole inference platform from Kubernetes-native pieces: vLLM for inference, the Endpoint Picker Plugin (EPP) for cache-aware routing, Gateway API for traffic management, KServe for model lifecycle, and Prometheus for observability. Every component is optional. Take what you need, skip what you don't.

But "which components do you need?" is exactly the question that trips people up, so that's the question I want to kill in this post. The answer hangs on five dimensions: model size, latency targets, traffic shape, GPU budget, and deployment environment. Run through those and you end up with a surprisingly small number of practical architectures. This post maps that space; the companion Choose Your Own Inference Stack interactive tool lets you poke at it dynamically.

2. The Decision Framework: Five Dimensions That Define Your Stack #

The most common mistake I see is treating component selection like a shopping list. Teams pick components off feature comparison charts and end up with a stack that's technically capable of everything and optimized for nothing. Stop doing that. Start from your constraints and work backward to components.

Five dimensions matter:

Model size is the load-bearing wall. A 7B parameter model on a single GPU is a completely different operational problem than a 70B model sharded across four GPUs. Model size sets your GPU topology, your minimum deployment unit, your scaling granularity, and your cost floor. Everything else is downstream of this.

Latency SLO shapes the whole architecture. If your p95 time to first token target is 2 seconds, you can run a unified serving pool and batch aggressively. If it's 200 milliseconds, you probably need prefill/decode disaggregation and prefix caching. And the difference between those two isn't incremental — it's structural.

Traffic pattern picks your routing strategy. Steady batch traffic wants max throughput. Bursty interactive traffic wants autoscaling that actually reacts. Agentic multi-turn traffic wants prefix-cache-aware routing that keeps sessions sticky to the pod holding their KV cache. A routing layer that doesn't know about any of this leaves 30-60% of your potential performance on the table.

GPU budget is the constraint that makes everything else real. There's always a theoretically better architecture that uses more GPUs. The practical question is: given N GPUs, what gets the most out of them? Budget also decides how much redundancy you can afford, whether dedicated prefill and decode pools are even on the table, and whether you need to share base model replicas across tenants with LoRA adapters.

Deployment environment sets your operational ceiling. An on-prem Kubernetes cluster with a three-person SRE team is a different world than a managed cloud with auto-provisioned node pools. The right architecture is the one your team can actually operate, monitor, and debug at 2 AM when it breaks.

Print that out. Tape it to your monitor. Next time someone asks "what inference stack should we use?" just walk them through those two questions. Nine times out of ten you land on Stage 3, and that's the right answer.

3. Component Deep Dives: What You Need and When #

llm-d is a composable platform, not a monolith. Each piece solves a specific operational problem. Here's when you need each one, and when you honestly don't.

vLLM Engine

When you need it: Always. This is the inference runtime. vLLM handles model loading, continuous batching, PagedAttention for KV cache management, tensor parallelism for multi-GPU serving, and quantized model execution. There's no llm-d deployment without vLLM.

The config decisions that matter: tensor parallelism degree (how many GPUs per replica), prefix caching (on by default in recent versions, and critical for multi-turn workloads), and quantization format (FP16, FP8, AWQ, GPTQ). Mostly your model size and GPU hardware pick these for you, and you set them once at deployment time in the serving config.

Endpoint Picker Plugin (EPP)

When you need it: The moment you have more than one replica. The EPP is the routing brain. It keeps a real-time view of every vLLM replica's state — queue depth, KV cache contents, loaded LoRA adapters, available capacity. A request comes in, the EPP scores each replica on cache hit potential, current load, and adapter availability, and sends it to the best one.

When you can skip it: One replica, no routing decision to make. But the second you scale to two or more, round-robin or random routing starts burning prefill compute for nothing. The EPP is usually the single highest-impact thing you can add after vLLM itself.

Gateway API

When you need it: When external traffic hits your stack. Gateway API gives you standard Kubernetes ingress with inference-aware extensions: per-model rate limiting, request prioritization, traffic splitting for canaries, TLS termination. It plugs into the EPP so routing decisions know about model-serving state.

When you can skip it: Internal-only workloads where services hit the endpoint directly over ClusterIP can put this off. Anything facing end users or partner APIs needs a real gateway though.

KServe / InferencePool

When you need it: When you want the model lifecycle managed for you. KServe handles model loading from object storage, health checking, readiness probing, rolling updates, and canary rollouts. InferencePool defines a logical group of serving pods that the EPP treats as one routing target. Together they're the layer that turns "a container running vLLM" into "a managed model endpoint."

Prometheus + Grafana

When you need it: In production. Always. vLLM exports a genuinely rich set of metrics: time to first token (TTFT), inter-token latency, tokens per second (TPS), queue depth, KV cache utilization, prefix cache hit rate, GPU memory usage, batch size distribution. Without these you're flying blind. You can't tune what you can't measure, and inference has enough moving parts that gut feel will absolutely mislead you.

HPA Autoscaler

When you need it: Variable traffic. The Horizontal Pod Autoscaler scales replica count off custom vLLM metrics, typically queue depth or pending request count. Scaling on CPU utilization is meaningless for GPU inference — you want inference signals: requests queuing, add capacity; GPUs idle, release them.

When you can skip it: Steady batch workloads with predictable traffic can just run a fixed replica count. But anything user-facing with variable traffic needs autoscaling, or you're either burning GPU hours or dropping requests.

Prefill/Decode Disaggregation

When you need it: Aggressive latency targets on long-context inputs. Prefill (chewing through the input prompt) is compute-bound. Decode (generating output tokens) is memory-bandwidth-bound. In a unified pool, long prefills block decodes and every concurrent request eats the latency spike. Disaggregation puts the two phases on dedicated GPU pools so neither blocks the other.

When you can skip it: Short prompts (under 1,000 tokens), small models (under 13B parameters), or relaxed latency targets (TTFT above 2 seconds) — in any of those cases the complexity isn't worth it. It roughly doubles your operational surface: two GPU pools, a KV cache transfer mechanism, and cross-pool routing logic to babysit.

4. Stack Patterns by Use Case #

Frameworks are nice, but concrete patterns are what people actually deploy. Here are five I keep seeing in production, with the architecture choices and the reasoning behind each.

Chatbot / Customer Support

Multi-turn conversation is the most common production use case, and it's where cache-aware routing pays off hardest. Every turn appends a few hundred tokens to a growing context window. Without prefix caching, every turn re-processes the whole conversation history from scratch. With it, only the new tokens need prefill.

Architecture: vLLM with prefix caching enabled, EPP with session-affinity scoring, Gateway API for external traffic, HPA scaling on queue depth. Models are typically 7B to 70B, often fine-tuned or LoRA-adapted for the support domain.

Expected performance: With prefix caching and cache-aware routing, TTFT on later turns drops 60-85% versus naive round-robin. For a 5-turn conversation on a 13B model, that's the difference between 800ms TTFT on turn five (no cache routing) and 120ms (with it).

Code Completion / Copilot

Code completion is latency-critical in a way chatbots just aren't. A developer typing expects suggestions in 100-200ms; anything slower feels broken. Prompts are short (a few hundred tokens of surrounding code), but request rate is brutal since every keystroke or pause can fire a completion request.

Architecture: Small, fast model (7B-13B), vLLM with speculative decoding if supported, EPP for load-aware routing (prefix caching matters less with short prompts), aggressive HPA scaling with fast pod startup. FP8 quantization to get the most throughput per GPU.

Key config: Set max_num_seqs high to handle the concurrency. Run gpu_memory_utilization at 0.90+ so the KV cache has room for concurrent requests. And tune max_model_len down if you know your prompts are short — that frees memory for more concurrent sequences.

RAG (Retrieval-Augmented Generation)

RAG traffic has its own personality: variable-length prompts assembled at runtime from retrieved documents, with a lot of prefix overlap when multiple users hit the same knowledge base. The system prompt, retrieval instructions, and frequently-retrieved docs all become shared prefixes across requests.

Architecture: vLLM with prefix caching, EPP with prefix-aware routing, models in the 13B-70B range depending on how much reasoning you need. Long context support matters a lot here — retrieved documents can push prompts to 8K-32K tokens.

Batch Processing

Batch workloads (document summarization, data extraction, classification pipelines) care about throughput and cost, not latency. The whole game is tokens per dollar per hour.

Architecture: vLLM with large batch sizes, load-aware routing (cache routing matters less since prompts are mostly unique), no autoscaling (fixed replica count sized for the job), aggressive quantization (FP8 or INT4) for throughput. Pipeline parallelism is worth a look for very large models.

Key config: Set max_num_seqs as high as your GPU memory allows. Stop caring about TTFT — aggregate throughput is the metric. Use offline batching with the vLLM batch API instead of the OpenAI-compatible streaming endpoint.

Real-time Classification / Extraction

Structured extraction and classification: small models, short prompts, and very little patience for latency. Think extracting entities from a customer message before routing it, scoring transaction risk in a payment pipeline, or parsing form fields out of scanned documents.

Architecture: Small model (1B-7B), single GPU per replica, FP8 quantization, EPP for load balancing across a lot of replicas, aggressive HPA scaling. These workloads look more like classic microservice traffic than LLM traffic: high concurrency, short requests, low tolerance for tail latency.

Expected performance: A 3B model on a single H100 with FP8 can handle classification requests with sub-50ms TTFT and spit out 5-10 output tokens in under 20ms. At that point the bottleneck stops being GPU compute and becomes network overhead and routing latency.

5. The Complexity-Performance Tradeoff #

Every component you add is operational surface area. New failure modes, new config knobs, new metrics to watch, new on-call runbooks to write. So the question is never "does this component improve performance?" — it almost always does. It's "does the improvement pay for the operational cost in my situation?"

The maturity curve

Teams work their way up through stages of stack complexity, and knowing which stage you're on saves you from both premature optimization and pointless suffering.

Stage 1, Single instance: One vLLM pod, one model, a Kubernetes Service in front. No routing intelligence, no autoscaling, no disaggregation. Fine for proof-of-concepts and low-traffic internal tools. Operational overhead: basically none. Performance ceiling: one GPU's worth of throughput.

Stage 2, Replicated with smart routing: Multiple vLLM replicas behind an EPP. Cache-aware routing, load balancing, basic health checking. For production traffic, this is where most teams should start. Adding the EPP to a replicated deployment typically improves TTFT 40-70% for multi-turn workloads at nearly zero extra GPU cost — the win comes from routing smarts, not more hardware.

Stage 3, Observable and scalable: Prometheus metrics, Grafana dashboards, HPA autoscaling on inference-specific metrics. This is the minimum viable production stack. Now you can actually see what's happening, react to traffic changes, and debug without guessing.

Stage 4, Disaggregated and specialized: Separate prefill and decode pools, KV cache transfer, specialized routing. This is for teams whose latency targets genuinely can't be met with a unified pool. Operational complexity roughly doubles, but the latency win on long-context workloads can be 3-5x.

Quantifying the tradeoff

Rough cost-benefit for each component:

6. Migration Paths: How to Evolve Your Stack #

The worst way to build an inference platform is designing the final architecture on day one and trying to deploy the whole thing at once. Go incremental: start with the simplest setup that meets your current needs, instrument it, measure it, and only add complexity when the measurements tell you to. Here's the progression, concretely.

Phase 1: Single vLLM instance

Trigger: You have a model and you need to serve it. Architecture: One vLLM pod, one Kubernetes Service, one model. What you gain: A working inference endpoint. What you risk: No redundancy, no scaling, no routing intelligence. Fine for development, staging, and low-traffic internal tools.

# Phase 1: Minimal vLLM deployment
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-server
spec:
  replicas: 1
  template:
    spec:
      containers:
      - name: vllm
        image: vllm/vllm-openai:latest
        args:
        - "--model"
        - "meta-llama/Llama-3.1-8B-Instruct"
        - "--enable-prefix-caching"
        resources:
          limits:
            nvidia.com/gpu: 1

Phase 2: Add EPP for cache-aware routing

Trigger: You scale to multiple replicas and notice TTFT varies wildly between requests, or you start serving multi-turn conversations and realize every turn is re-computing the full context. What you gain: The EPP tracks KV cache state across all replicas and routes each request to the pod with the longest matching cached prefix. For multi-turn workloads this is usually the single biggest performance win available — often bigger than upgrading GPUs. What you risk: The EPP is one more component that has to stay healthy. If it goes down you fall back to default Kubernetes routing — still functional, just not cache-aware.

Phase 3: Add monitoring and alerting

Trigger: You're in production and you need to answer things like "why was latency high at 3 PM?" or "are we gonna need more GPUs next month?" What you gain: Visibility into TTFT distributions, cache hit rates, queue depths, GPU utilization, and throughput trends. You can alert on p95 TTFT, queue depth thresholds, and GPU memory utilization. What you risk: Prometheus and Grafana are mature, well-understood tools, but they still need storage, retention policies, and dashboard upkeep. Budget 4-8 hours for initial setup and 1-2 hours a week ongoing.

Phase 4: Add autoscaling

Trigger: Your traffic is variable enough that a fixed replica count either wastes GPUs off-peak or drops requests at peak. What you gain: HPA scales replicas off vLLM-exported metrics (queue depth, pending requests). You stop paying for idle GPUs at night and stop dropping requests at lunch. What you risk: GPU autoscaling is slower than CPU autoscaling. New vLLM pods have to load model weights, which takes 30 seconds to several minutes depending on model size. Your scaling policy has to account for that startup time with sensible stabilization windows and predictive scaling.

Phase 5: Add disaggregation

Trigger: Your measured p95 TTFT exceeds your SLO, and profiling shows long prefills blocking decodes on shared GPUs. This usually shows up with models above 30B parameters, prompts above 2,000 tokens, and TTFT targets below 500ms. What you gain: Prefill and decode run on separate GPU pools — prefill pods tuned for compute throughput, decode pods for memory bandwidth, neither blocking the other. What you risk: You're now managing two GPU pools, KV cache transfer between them, and cross-pool routing. Operational complexity roughly doubles. Only do this with real, measured evidence that unified serving can't hit your SLOs.

Phase 6: Add multi-tenancy

Trigger: A second team wants their fine-tuned adapter on the same infrastructure, or you need isolated model endpoints for different customers or orgs. What you gain: LoRA-aware routing lets multiple fine-tuned adapters share base model weights on the same GPU. The EPP routes to replicas that already have the requested adapter loaded, stacking adapter awareness on top of cache awareness. Namespace isolation gives you security boundaries between tenants. What you risk: Multi-tenant scheduling makes routing decisions harder. Adapter loading has a cold-start penalty (typically 1-5 seconds). And you'll need policies for adapter eviction, priority, and resource quotas.

7. Migration Horror Stories: Lessons Learned the Hard Way #

Theory is clean. Production is not. That migration path looks smooth on paper, but every team that's walked it has hit sharp edges. These three scenarios are composites of real incidents — the kind of failures that never show up in architecture diagrams but absolutely show up on your pager.

Horror Story #1: The Silent KV Cache Regression

Horror Story #2: The Configuration Drift Deploy

Horror Story #3: The LoRA Adapter Mismatch

8. Deep Comparison with Alternative Stacks #

I'll be blunt here. If you're running on Kubernetes, use llm-d. Full stop. Nothing else gives you cache-aware cross-replica routing, LoRA multi-tenancy, and prefill/decode disaggregation as composable, Kubernetes-native pieces. But you deserve to know why I think that, so let's walk the alternatives honestly and in depth.

TensorRT-LLM + Triton Inference Server

NVIDIA's native stack, and the fastest single-GPU option on NVIDIA hardware. TensorRT-LLM compiles models into optimized CUDA kernels with aggressive operator fusion, and Triton is a mature model-serving framework with dynamic batching, model versioning, and ensemble pipelines.

Where it wins: Raw inference speed on NVIDIA GPUs, especially models that benefit from FP8 quantization and custom CUDA kernels. The gap is real — typically 10-25% more throughput per GPU than vLLM for supported models. Triton also has a genuinely mature model management story: versioning, A/B testing, ensemble pipelines that chain multiple models. If you're deep in the NVIDIA ecosystem with a dedicated MLOps team, this is the tightest integration with NVIDIA hardware you can get.

Where it falls short: TensorRT-LLM requires model compilation, which adds minutes to hours to the deploy pipeline and makes fast model iteration painful. It also means no hot-swapping models or adapters — every LoRA adapter change is a new compilation cycle. Triton's routing isn't KV-cache-aware; it does dynamic batching inside a single instance but knows nothing about cross-replica cache state. The stack is NVIDIA-only, which matters more every quarter as AMD GPUs (the MI300X especially) get cheaper and easier to get in the cloud. Multi-tenant LoRA serving is manual config, not dynamic routing. If you want TRT-LLM speed plus smart routing, you're building a lot of custom infrastructure yourself.

Community and ecosystem: NVIDIA puts real resources behind TensorRT-LLM, but it's developed NVIDIA's way. Community contributions are thin compared to open-source alternatives, and issue resolution depends on NVIDIA's internal priorities. The Triton ecosystem is mature with good docs and examples, but it's built around NVIDIA's hardware assumptions.

How it holds up in production: Solid for single-instance deployments on NVIDIA hardware. Shakier for multi-replica orchestrated deployments, where you're bringing your own routing, scaling, and cache management. There's a Triton Kubernetes operator, but it's basic orchestration, not inference-aware routing.

SGLang

SGLang is a seriously good inference engine with excellent prefix caching (RadixAttention) and structured generation support. It especially shines on constrained decoding: JSON output, grammar-guided generation, fill-in-the-middle.

Where it wins: Structured output performance, single-instance prefix caching, and developer ergonomics for prompt programming. RadixAttention is arguably the best single-instance prefix caching implementation out there — cached prefixes live in a radix tree, so longest-prefix matching is fast and cheap. If your workload is mostly single-instance with heavy structured generation (JSON schema enforcement, regex-constrained output, grammar-guided decoding), SGLang can meaningfully beat vLLM. The programming model is genuinely elegant too, with first-class prompt composition, branching, and constrained generation.

Where it falls short: SGLang is basically a single-process engine. No distributed orchestration, no cross-replica cache-aware routing, no Kubernetes-native deployment primitives, no autoscaling. Multi-replica production means building the routing and scaling layer yourself or bolting on an external orchestrator. There's no EPP equivalent for cross-replica cache coordination — scale to multiple SGLang instances and each one keeps its own independent radix tree, with no way to route a request to the instance that actually holds the cached prefix. So you lose RadixAttention's main advantage the moment you need a second replica. Great engine, no platform.

Community and ecosystem: SGLang has a lively research community with regular academic contributions and active development. But it's smaller than vLLM's, and the production tooling (Helm charts, Kubernetes operators, monitoring integrations) is less mature. Most production SGLang deployments I've seen run on custom scripts, not standardized infrastructure-as-code.

How it holds up in production: Great for single-instance workloads, especially structured generation. Rough for multi-replica deployments — no distributed orchestration, no routing intelligence, thin ops tooling, so you're signing up for a lot of custom engineering.

Ray Serve + vLLM

Ray Serve wraps vLLM in Ray's distributed computing framework, adding autoscaling, model multiplexing, and hooks into the wider Ray ML ecosystem (Ray Train, Ray Data). If your team is already invested in Ray for training and data processing, this is the natural move.

Where it wins: Integration with Ray's ML pipeline tools, model composition (chaining models, pre/post-processing), and autoscaling across heterogeneous resources. If you already run Ray for distributed training, data processing, or hyperparameter tuning, Ray Serve gives you one control plane across the whole ML lifecycle, training through serving. Model multiplexing lets you serve multiple models from a single deployment, and Ray's resource management handles mixed GPU types in one cluster. The autoscaling is mature too — custom metrics, smart placement strategies, the works.

Where it falls short: Ray's distributed runtime costs you real memory and CPU. A Ray cluster means head nodes, worker nodes, the Global Control Store (GCS), and the Ray dashboard, all eating resources that could be serving inference. More importantly, Ray Serve's routing isn't inference-aware. It doesn't know about KV cache state, prefix overlap, or LoRA adapter locality — routing runs on generic load metrics (queue depth, replica count), not the signals that actually determine latency. And if you're not already a Ray shop, the learning curve and ops overhead are steep. You're buying a general-purpose distributed computing framework when what you wanted was an inference-aware routing layer. Ray's failure modes are gnarly too; a GCS failure can cascade across the whole cluster in ways that are hard to diagnose without deep Ray expertise.

Community and ecosystem: Ray has a big, active community backed by Anyscale, extensive docs, plenty of production deployment examples. But the intersection of "Ray Serve" and "LLM inference at scale" is still pretty niche. Most production Ray Serve + vLLM deployments are at places that were already Ray shops — not places that picked Ray for inference serving.

How it holds up in production: Well, if your team already has Ray expertise and infrastructure. Otherwise it's a grind. Ray ops skills (debugging GCS issues, placement groups, tuning autoscaling policies) are hard-won and completely orthogonal to inference serving expertise. KubeRay gets you Kubernetes integration, but that's another layer of complexity stacked on Ray's own.

Triton Inference Server (standalone)

Triton can also run standalone without TensorRT-LLM, serving models in ONNX, TensorFlow SavedModel, or PyTorch formats. In that mode it's a general-purpose model server with dynamic batching, model versioning, and ensemble support.

Where it wins: Multi-framework serving. If you're serving a mix of LLMs, vision models, embedding models, and traditional ML from one piece of infrastructure, Triton's multi-backend support is genuinely useful. It can run an ONNX embedding model, a PyTorch classifier, and a vLLM-based LLM in the same deployment, chained into an ensemble pipeline. The model repository pattern (versioned model artifacts in shared storage) gives you clean lifecycle management, and health checking, readiness probing, and model loading/unloading are all well done.

Where it falls short: For LLM inference specifically, standalone Triton gives you nothing over vLLM and takes a few things away. No PagedAttention, no continuous batching for autoregressive generation, no prefix caching. Its dynamic batching was built for fixed-shape tensor inputs, not variable-length token sequences. For LLM workloads, Triton ends up being a container that runs vLLM or TRT-LLM as a backend — a layer of indirection with no inference-specific routing intelligence added. The Kubernetes operator does basic pod management, not cache-aware routing or LoRA-aware scheduling.

Community and ecosystem: Triton has a big user base in traditional ML serving (recommendation systems, vision, tabular). The LLM-specific community is smaller and mostly overlaps with the TRT-LLM crowd. Docs are thorough but assume you know NVIDIA's model repository structure and config format.

How it holds up in production: Fine for mixed-model serving. For LLM-only workloads it's overhead without payoff — if you don't need multi-framework support, wrapping vLLM in Triton adds complexity for basically nothing.

Ollama

Ollama is the right tool for local dev, experimentation, and single-user desktop inference. The developer experience is honestly lovely: one command to download and serve a model, a simple API, broad model support.

Where it wins: Developer experience, local dev workflow, ease of setup.

Where it falls short: Ollama was never meant for production serving. No multi-replica support, no cache-aware routing, no autoscaling, no GPU sharing, no LoRA adapter routing, none of the monitoring and observability plumbing production needs. Running Ollama in production is like running SQLite as your production database — it works until it doesn't, and when it doesn't, it's sudden. Don't do it.

llm-d

llm-d's real differentiator is that it treats inference serving as a distributed systems problem, not just a single-GPU optimization problem. The EPP's cache-aware routing, the Gateway API integration, and the Kubernetes-native deployment model are all aimed squarely at the gap between "inference engine" and "inference platform."

Where it wins: Cache-aware cross-replica routing, LoRA-aware multi-tenant serving, Kubernetes-native deployment and scaling, a composable design that lets you adopt pieces incrementally, and multi-vendor GPU support (NVIDIA and AMD).

Where it has limits: It requires Kubernetes, and if your team doesn't have Kubernetes chops, that learning curve is real. Raw single-GPU speed is vLLM's — excellent, but not the absolute fastest on NVIDIA hardware (TensorRT-LLM edges it out on some workloads). And the project is young (CNCF Sandbox), so the third-party integration ecosystem is still filling in. It's filling in fast though, and the CNCF backing means it's not going anywhere.

9. Decision Matrix: Choosing Your Stack at a Glance #

The deep comparison above covers the nuance, but sometimes you just need one table you can screenshot and drop in a Slack thread. So here it is — each stack scored across the dimensions that matter most in production.

When to choose each stack

Every stack has a sweet spot. Here's the whole comparison boiled down to cards you can skim.

10. Total Cost of Ownership Across Stacks #

Picking an inference stack isn't just a performance decision — it's a cost decision. Total cost of ownership means GPU compute, engineering time for setup and maintenance, day-to-day ops overhead, and whatever licensing or lock-in you're signing up for. Here's how the major stacks compare on a representative production deployment: a 70B model serving 100 requests per second with a 500ms p95 TTFT target.

11. Decision Checklist for Enterprises #

Before you commit to an architecture, work through these ten questions. Each one maps to a specific decision. And if you can't answer one — that's a finding in itself. It means you've got homework to do before you can decide anything.

• 1 What model(s) will you serve? Sets your GPU memory requirements, tensor parallelism degree, and minimum deployment cost. A 7B model on one GPU is a totally different operational problem than a 70B model across four.
• 2 What are your latency SLOs? Tells you whether you need unified serving (TTFT > 1s), cache-aware routing (TTFT 200ms-1s), or disaggregation (TTFT < 200ms on long contexts). Be specific: p50, p95, p99.
• 3 How many teams or tenants will share the infrastructure? Single-tenant is easy. Multi-tenant means LoRA-aware routing, namespace isolation, resource quotas, and chargeback. Every tenant you add makes routing a little harder.
• 4 What is your GPU budget? Sets replica count, redundancy, and whether dedicated prefill/decode pools are even affordable. Also how hard you should chase efficiency wins like quantization and cache-aware routing.
• 5 Cloud or on-premise? Cloud gets you elastic GPU provisioning at a higher per-hour cost. On-prem is cheaper per hour but you're doing capacity planning and waiting on hardware procurement. Hybrid adds networking headaches.
• 6 What is your ops team's Kubernetes maturity? llm-d is Kubernetes-native, which is a strength if your team lives in Kubernetes and a barrier if they do not. Be honest about this. The best architecture is one your team can operate at 2 AM.
• 7 Do you need LoRA adapters? If multiple fine-tunes share a base model, LoRA adapters with dynamic routing can cut GPU cost 5-10x versus serving separate model instances. This decides whether you need LoRA-aware routing in the EPP.
• 8 What is your availability target? 99.9% needs at least 2 replicas with health checking and failover. 99.99% needs geographic redundancy or multi-zone. Every extra nine costs you, in ops and in dollars.
• 9 Do you have existing monitoring infrastructure? Already running Prometheus and Grafana? Wiring in vLLM metrics is easy. Starting from scratch? Budget 1-2 weeks for setup, dashboard design, and alert tuning.
• 10 What is your scaling profile? Steady traffic: fixed replicas. Predictable peaks (business hours, campaign launches): scheduled scaling. Unpredictable spikes: reactive HPA with sensible stabilization windows and startup time buffers.

12. Conclusion #

If you take one thing from this post: there's no universally correct inference stack. But there is a very short list of practical architectures that cover the vast majority of production use cases. Model size plus latency target gets you 80% of the way there. The other three dimensions — traffic shape, GPU budget, ops maturity — just refine the details.

If you're on Kubernetes, llm-d gives you the pieces to build the right architecture from Kubernetes-native primitives. Start with vLLM and the EPP. Add Prometheus. Add autoscaling. That's Stage 3, and it handles most production workloads. Only add disaggregation when your measurements demand it. Every step should be triggered by evidence, not aspiration.

The Choose Your Own Inference Stack interactive tool walks you through each dimension, shows how your choices interact, and hands you an architecture recommendation. It's a fun way to explore the decision space before you commit to anything.

Want to dig deeper into the project and community:

• Interactive tool: Choose Your Own Inference Stack
• llm-d on GitHub: github.com/llm-d/llm-d
• Documentation: llm-d.ai
• CNCF Slack: #llm-d on CNCF Slack
• Content Hub: llm-d Content Hub

Stop planning. Start deploying.