Deploy llm-d on Kubernetes in 5 minutes: from zero to intelligent inference routing

If you're serving large language models (LLMs) on Kubernetes today, your inference stack probably looks something like this: vLLM or TGI behind a standard load balancer, with requests distributed round-robin across your GPU-backed replicas. It works. Until you start scaling and realize that every follow-up message in a multi-turn conversation is recomputing prefixes that were already computed seconds ago on a different pod.

Standard round-robin load balancing: follow-up requests land on random pods and recompute from scratch.

That's the problem llm-d solves. It's an open source, Kubernetes-native inference routing layer that sits in front of your model servers and makes intelligent decisions about where to send each request. Prefix-cache-aware routing, load-aware balancing, predicted-latency optimization. All built on top of the Gateway API InferencePool, so you're not locked into any proprietary routing setup.

llm-d routes follow-up requests to the pod that already has the prefix cached.

In this post, we'll walk through a full llm-d deployment from scratch in under five minutes. No slides, no theory. Just a working stack on Red Hat OpenShift with a GPU, a model, and a router.

Four steps: Gateway API CRDs, Gateway Provider, Model Server, Router.

What you'll need

Before you get started, you'll need:

• An OpenShift or Kubernetes cluster with at least one GPU node
• Helm 3 installed locally
• oc or kubectl CLI configured and logged in
• A model to serve: We're using Qwen3-0.6B here for speed, but the same setup works with 70B+ models

Step 1: Install the Gateway API CRDs

llm-d's routing layer is built on the Kubernetes Gateway API and the Inference Extension API. You need both sets of Custom Resource Definitions (CRDs) installed before anything else.

oc apply -k "https://github.com/kubernetes-sigs/gateway-api/config/crd/?ref=v1.5.1"
oc apply -k "https://github.com/kubernetes-sigs/gateway-api-inference-extension/config/crd/?ref=v1.5.0"

These CRDs give you access to the InferencePool and InferenceModel resources, which are how the llm-d router discovers and communicates with your model servers.

Step 2: Deploy the gateway provider

llm-d supports multiple gateway providers, including Envoy and AgentGateway. For this walkthrough, we're using AgentGateway because it's a single Helm install.

helm install agentgateway oci://ghcr.io/agentgateway/charts/agentgateway \
  --version v1.1.0 \
  --namespace agentgateway-system \
  --create-namespace

Then apply the gateway configuration from the llm-d repository:

oc apply -k guides/recipes/gateway/agentgateway/

At this point, the gateway is running and ready to accept routing rules.

Step 3: Deploy the model server

The model server is standard vLLM running behind an llm-d deployment configuration. The llm-d repository includes nine pre-tested deployment recipes called well-lit paths that cover common hardware and model configurations. We're using the optimized-baseline path here.

oc apply -f vllm-deploy.yaml

The deployment YAML configures vLLM with the appropriate GPU tolerations, resource requests, and volume mounts for cache directories. It also includes the label llm-d.ai/guide: optimized-baseline, which is how the InferencePool selector finds the pod.

The model takes roughly 60 to 90 seconds to download and load into GPU memory. Once vLLM reports Application startup complete in the logs, it's ready to serve.

Step 4: Deploy the router

The router is the component that makes llm-d more than just vLLM on Kubernetes. It's deployed as a Helm chart:

helm install optimized-baseline \
  oci://ghcr.io/llm-d/charts/llm-d-router-standalone-dev \
  -f guides/recipes/router/base.values.yaml \
  -f guides/optimized-baseline/router/optimized-baseline.values.yaml \
  -n llm-d-demo --version v0

After this, you should see two pods running: the vLLM model server and the llm-d router (EPP). You can verify the router has discovered the model server by checking the InferencePool:

oc get inferencepool -n llm-d-demo

That's it. Four steps. The full llm-d stack is deployed and ready to route inference requests.

Full stack: Client → AgentGateway → llm-d Router (EPP) → InferencePool → vLLM model server.

Sending requests: direct vs. routed

To understand what the router adds, it helps to see both paths side by side.

Direct to vLLM

Port-forward to the model server and send a standard OpenAI-compatible request:

oc port-forward deploy/vllm-decode 8000:8000 -n llm-d-demo &
curl -s http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"Qwen/Qwen3-0.6B",
       "messages":[{"role":"user","content":"Explain Kubernetes in three sentences."}],
       "max_tokens":150}' | python3 -m json.tool

This works exactly as expected. You get a response, and it's identical to what you'd see from any standalone vLLM deployment.

Through the llm-d router

Now send the same request through the router:

oc port-forward svc/optimized-baseline-epp 8080:80 -n llm-d-demo &
curl -s http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"Qwen/Qwen3-0.6B",
       "messages":[{"role":"user","content":"Explain Kubernetes in three sentences."}],
       "max_tokens":150}' | python3 -m json.tool

The response looks the same. But under the hood, the router has cached the prefix of this conversation. When you send a follow-up message, the router knows which backend already has the key-value (KV) cache for this conversation and routes the request there instead of recomputing it from scratch.

First request creates KV cache on Pod 1. Follow-up is routed to the same pod, skipping prefix recomputation.

Why this matters at scale

With a single replica, the benefit is subtle. But in production, where you might have 10, 20, or more model server replicas handling hundreds of concurrent multi-turn conversations, the difference is significant.

Without llm-d, each follow-up message could land on any random replica and recompute the entire conversation prefix. With llm-d's prefix-cache-aware routing, the follow-up goes to the replica that already has the context cached. That means less GPU compute per request, higher throughput on the same hardware, and lower cost per token served.

What's next

This walkthrough used a 0.6B model on a single GPU to keep things quick. But the same deployment pattern scales to much larger configurations:

• Disaggregated prefill and decode: Separate the compute-intensive prefill phase from the memory-bound decode phase across different worker pools
• Multi-LoRA serving: Serve multiple fine-tuned adapters from a single base model, with the router directing requests to the replica that has the right adapter loaded
• FP8 quantization: Run 70B+ models on fewer GPUs using production-grade quantization from Red Hat AI

llm-d is an open source project under the Cloud Native Computing Foundation (CNCF) sandbox. The full source, documentation, and well-lit path deployment guides are available in the llm-d repository on GitHub.