What Running LLM Inference in Production Actually Teaches You

Production LLM Inference is a Different Animal

It's 02:47 on a Tuesday. Your phone buzzes. PagerDuty. TTFT p99 has crossed 12 seconds -- your SLO is 500 milliseconds. You open the dashboard from bed. The KV cache hit rate chart looks like a cliff face. Error rate is climbing at a slope that makes your stomach drop. Forty-seven Slack messages in the #incidents channel, and the most recent one says: "Are we rolling back?"

This is the moment where everything you thought you knew about deploying models gets stress-tested against reality. And if you are reading this before that moment arrives, you are already ahead.

The moment you deploy a model behind an API, you are in the critical path of user-facing applications. Latency budgets shrink from hours to milliseconds. Every incoming request is different: variable input lengths, unpredictable output lengths, wildly varying computational cost. Your system has to make real-time scheduling decisions thousands of times per second, and each decision carries direct user impact.

The mental model shift is real. Inference serving shares more DNA with database query optimization than with training orchestration. You are managing scarce resources (GPU memory, compute bandwidth, network), handling concurrent access to shared state (the KV cache), making admission control decisions under load, and trying to maintain tail-latency SLOs while the workload pattern shifts constantly.

And then there is the operational reality. Model weights are 140GB or more. Loading them takes minutes. GPU memory fragments. Caches evict at the worst possible moment. Health checks lie. Config changes race. The failure modes are unlike anything you encounter in traditional web services, and most teams only discover them the hard way -- in production, at scale, under pressure.

This post collects the lessons we have learned running llm-d in production since Red Hat AI 3 GA in October 2025. These are the scenarios that caught us off guard, the patterns that saved us, the observability practices that let us sleep at night, and the hard-won operational knowledge we wish someone had written down before we needed it.

12 Production Scenarios That Will Break Your Assumptions

Every team that moves from "it works on my GPU" to "it runs in production" hits walls they did not see coming. These are twelve real production scenarios, each with a root cause analysis and the fix that resolved it. They are ordered roughly by how early in your production journey you are likely to encounter them.

Monitoring and Observability for LLM Inference

Traditional web service monitoring will betray you. We learned this when our dashboards showed green across every panel while 43% of our responses were being silently truncated. GPU utilization alone tells you almost nothing useful -- a GPU at 60% utilization might be serving beautifully, or it might have a queue 40 requests deep because every request is a 128K-context monster.

The first time we hit a KV cache eviction storm, we spent 20 minutes staring at GPU utilization graphs that looked completely normal. Queue depth was the metric that would have told us what was coming -- 4 minutes before TTFT exploded. We did not have it on the dashboard yet. Now it is the first panel.

The Metrics That Actually Matter

The Alerts You Need Day One

We had none of these alerts during our first month in production. Every one was added after an incident. Save yourself the pages.

The Dashboards That Save You at 3 AM

GPU Health Monitoring: The Metrics Your Dashboard Is Missing

We spent our first three months watching GPU utilization and memory usage. Those are fine metrics. They are also the equivalent of checking a car's speedometer while ignoring the engine temperature, oil pressure, and tire pressure. The GPU has dozens of health signals that predict failures hours or days before they become incidents. Here is what we learned to watch after each one bit us.

ECC Error Monitoring

GPUs have error-correcting code (ECC) memory. Correctable errors are normal -- a few per hour is fine. But correctable errors are a leading indicator of uncorrectable errors, and uncorrectable errors corrupt your model state mid-inference. We lost an entire pod to this because nobody was trending ECC error rates.

The pod had been reporting 2 correctable ECC errors per hour for weeks. On a Thursday afternoon, the rate climbed to 50/hour. By Friday morning it was 200/hour. Saturday at 14:22 UTC, the first uncorrectable error hit during a decode operation. The model produced garbage output for 3 requests before the pod crashed. We did not have ECC trending on any dashboard.

# ECC error rate trending -- predict GPU failure before it happens
rate(DCGM_FI_DEV_ECC_SBE_VOL_TOTAL[1h]) # Correctable error rate

# Critical: any uncorrectable errors
increase(DCGM_FI_DEV_ECC_DBE_VOL_TOTAL[5m]) > 0

Clock Throttling Detection

GPUs throttle their clock speed for three reasons: power limits, thermal limits, and reliability limits. Each one has a different operational response. We had a pod running at 60% of expected throughput for two days before someone realized the GPU was power-throttling because a neighboring workload on the same node was consuming more power than expected.

PCIe Bandwidth Monitoring

PCIe link degradation is rare but catastrophic when it happens. A link that drops from x16 to x8 (or worse, x1) halves (or destroys) your CPU-GPU transfer bandwidth. This affects model loading, KV cache operations in non-disaggregated mode, and any operation that moves data between host and device. We caught one of these only because a pod's model load time jumped from 90 seconds to 6 minutes.

# Check PCIe link width and speed
nvidia-smi --query-gpu=pcie.link.width.current,pcie.link.width.max,pcie.link.gen.current,pcie.link.gen.max --format=csv

# Expected for A100: Gen4 x16. If you see Gen3 or x8, investigate immediately.

Temperature Trending and Thermal Prediction

GPU temperature does not just affect performance through throttling. Sustained high temperatures accelerate electromigration and reduce component lifetime. More practically, temperature trends predict thermal throttling before it happens. We track temperature over 24-hour windows and alert when the trend predicts throttle temperature within 2 hours at current rate.

# Predict thermal throttling: linear regression on temperature
predict_linear(DCGM_FI_DEV_GPU_TEMP[2h], 7200) > 83

NVLink Health for Multi-GPU Inference

If you are running tensor-parallel inference across multiple GPUs (common for 70B+ models), NVLink health is on your critical path. NVLink errors cause retransmissions that add microseconds per operation -- and with millions of operations per forward pass, those microseconds compound into milliseconds of decode latency.

Prometheus Queries for Inference Metrics

If you are running Prometheus and Grafana (and you should be), these are the PromQL queries that belong on your inference dashboards. Each query targets a specific operational concern described in the scenarios above. We run every one of these in production. The ones we did not have on day one are the ones that correspond to our worst incidents.

TTFT p99 Tracking

This is the query that was missing from the dashboard during the ConfigMap cascade (War Story #1). We were watching GPU utilization when we should have been watching TTFT. Now it is the largest panel on the main dashboard, refreshing every 10 seconds.

# TTFT p99 across the fleet, 5-minute window
histogram_quantile(0.99,
  sum(rate(
    vllm_time_to_first_token_seconds_bucket[5m]
  )) by (le, pod)
)

Inter-Token Latency

The NUMA misplacement (Scenario #2) would have been caught on day one if we had been tracking per-pod ITL. One pod with 38ms ITL against a fleet average of 22ms -- it sticks out immediately on a per-pod breakdown.

# ITL p95 per pod -- detects decode slowdowns
histogram_quantile(0.95,
  sum(rate(
    vllm_inter_token_latency_seconds_bucket[5m]
  )) by (le, pod)
)

Queue Depth Per Pod

# Current queue depth -- leading indicator of saturation
vllm_num_requests_waiting{namespace="inference"}

# Queue depth ratio vs fleet average -- detects hot-spotting
vllm_num_requests_waiting
  / on() group_left()
  avg(vllm_num_requests_waiting)

KV Cache Utilization and Hit Rate

The cache hit rate query below is the one that would have predicted the eviction storm (Scenario #1) four minutes before TTFT spiked. When you see hit rate dropping, start shedding load. Do not wait for TTFT to confirm.

# KV cache utilization -- alert at 85%
vllm_gpu_cache_usage_perc{namespace="inference"}

# Prefix cache hit rate over 5 minutes
rate(vllm_prefix_cache_hits_total[5m])
  / (
    rate(vllm_prefix_cache_hits_total[5m])
    + rate(vllm_prefix_cache_misses_total[5m])
  )

GPU Memory and Fragmentation

# GPU memory utilization from DCGM
DCGM_FI_DEV_FB_USED / (DCGM_FI_DEV_FB_USED + DCGM_FI_DEV_FB_FREE)

# Custom fragmentation ratio (requires custom exporter)
(gpu_memory_free_bytes - gpu_memory_largest_contiguous_free_bytes)
  / gpu_memory_free_bytes

Error Rate by Type

# finish_reason distribution -- detect silent truncation
sum(rate(
  vllm_request_success_total{finish_reason="length"}[10m]
)) / sum(rate(
  vllm_request_success_total[10m]
))

# Request error rate by error type
sum by (error_type) (
  rate(vllm_request_failure_total[5m])
)

Capacity Planning for Production Inference

We sized our first cluster for average load. It lasted one business day before the 10 AM traffic spike blew through every SLO we had set. Sizing a GPU cluster for inference is not like sizing a web application where you estimate capacity from request rate and average response time. Inference workloads have a complex resource profile that depends on model size, context length distribution, and the specific trade-offs between throughput and latency your SLO requires.

GPU Memory Budget Breakdown

Before you can estimate pod count, you need to understand how GPU memory is consumed. For a typical 70B parameter model on an 80GB A100, the memory budget looks roughly like this:

The critical insight: KV cache is the variable-cost component that determines your concurrent capacity. Every additional concurrent request with a 4K context consumes roughly 256MB of KV cache memory (for a 70B model). With 35GB available for KV cache, you can serve approximately 135 concurrent 4K-context requests -- or just 8-9 concurrent 64K-context requests. Context length distribution is the dominant factor in capacity planning.

We learned this the hard way when a single team switched their application from 2K-context queries to 32K-context RAG queries. Their traffic volume did not change. Their GPU allocation request did not change. But their per-request KV cache consumption went up 16x overnight, and they saturated a pod that had been running at 30% utilization the day before.

Compute Budget

The compute side has two distinct profiles that align with disaggregated serving:

• Prefill is compute-bound. FLOPs required scale linearly with input token count. A 4K prefill on a 70B model takes approximately 30ms on an A100; a 128K prefill takes approximately 960ms. Prefill throughput is limited by available TFLOPS.
• Decode is memory-bandwidth-bound. Each decode step reads the full KV cache from HBM. Decode throughput is limited by HBM bandwidth (2 TB/s on A100, 3.35 TB/s on H100). More concurrent decodes share bandwidth, reducing per-request throughput.

Rules of Thumb for Pod Count

These are starting-point estimates, not final answers. Your actual capacity will depend on your specific workload distribution, SLO requirements, and hardware configuration.

• Base capacity: Start with 1 pod per 100-150 requests per minute for short-context workloads (under 4K tokens average input). Reduce proportionally for longer contexts.
• Burst headroom: Provision 30-40% additional capacity above your average load to handle traffic spikes without SLO violations. New pods take minutes to spin up, so your steady-state fleet must absorb bursts.
• Maintenance windows: Plan for at least 10% of fleet capacity to be unavailable at any time for rolling updates, defragmentation, and pod recycling.
• Disaggregated ratio: For disaggregated deployments, a typical starting ratio is 1 prefill pod per 3-4 decode pods, tuned based on your workload's input-to-output token ratio.

When to Scale: Leading vs. Lagging Indicators

We wasted our first month scaling on lagging indicators. By the time GPU utilization crossed our threshold, the queue was already 40 requests deep and TTFT had blown past the SLO. The fix was learning to distinguish between metrics that predict load and metrics that confirm it.

Leading indicators -- these tell you to scale before users notice:

Lagging indicators -- these confirm you are already in trouble:

Autoscaling Strategies Compared

We tried three autoscaling approaches before finding the right one. Here is what we learned about each:

We landed on KEDA with EPP-reported queue depth as the primary signal and cache hit rate as a secondary guard rail. The scaling rules: scale up when average queue depth exceeds 8 for 2 minutes, scale down when average queue depth is below 2 for 15 minutes (conservative), never scale down more than 1 pod per 5-minute window.

Cost Modeling: GPU-Hours per Million Tokens

GPU-hours per million tokens is the metric that connects your inference costs to your business metrics. We track this per model, per operation (prefill vs decode), and per hardware generation. The numbers below are from our production measurements. Your mileage will vary based on quantization, batching efficiency, and workload mix.

Key takeaways from this data: Decode is 2.5-3x more expensive than prefill per token across all model sizes, because decode is memory-bandwidth-bound and processes one token at a time. H100 provides roughly 1.8x cost improvement over A100, primarily from higher HBM bandwidth (3.35 TB/s vs 2 TB/s). Larger models scale worse than linearly because they require more tensor-parallel GPUs, adding communication overhead.

Multi-Model Fleet Management

Once you move past a single model, fleet management becomes a resource allocation problem. We run 7B, 13B, and 70B models on the same GPU pool, and the scheduling decisions are non-trivial.

GPU partitioning considerations: NVIDIA MPS (Multi-Process Service) and MIG (Multi-Instance GPU) both allow multiple models on a single GPU, but they have very different trade-offs for inference workloads.

• MPS shares the full GPU across processes. Good for small models that do not individually saturate compute or memory bandwidth. Bad for large models or latency-sensitive workloads because contention is unpredictable.
• MIG partitions the GPU into isolated instances with guaranteed compute and memory. Better isolation than MPS, but each partition gets a fraction of HBM bandwidth. A 7B model on a 1/7 MIG partition of an A100 gets roughly 285 GB/s memory bandwidth instead of 2 TB/s. Decode throughput suffers proportionally.
• Dedicated GPUs per model is the approach we use for production workloads with latency SLOs. No contention, no surprises, no shared failure domains. More expensive, but you actually hit your SLOs.

Model priority classes: Not all models are equal. We assign priority classes that govern preemption behavior during resource contention:

• Critical -- customer-facing chat models. Never preempted. Guaranteed GPU allocation.
• Standard -- internal RAG pipelines. Can be temporarily throttled (queue depth limit) but not evicted.
• Bulk -- batch processing, evaluation runs. Can be preempted entirely during peak hours and rescheduled during off-peak.

Disaggregated Ratio Tuning

In a disaggregated deployment, the ratio of prefill pods to decode pods directly determines your system's throughput profile. Get this ratio wrong and you will either bottleneck on prefill (long TTFT, decode GPUs sitting idle) or bottleneck on decode (fast TTFT, slow ITL, prefill GPUs underutilized).

How to find the optimal ratio:

1. Measure your workload's input:output token ratio. If your average request has 2,000 input tokens and generates 500 output tokens, your I/O ratio is 4:1. Higher ratios mean more prefill-heavy workloads.
2. Start with the rule of thumb: 1 prefill pod per 3-4 decode pods for typical conversational workloads (I/O ratio ~2:1 to 4:1). For RAG workloads with long inputs and short outputs (I/O ratio 10:1+), you may need 1:2 or even 1:1.
3. Monitor the imbalance signals: If prefill queue depth is consistently higher than decode queue depth, add prefill pods. If decode ITL is climbing while prefill pods show low utilization, add decode pods.
4. Adjust weekly: Workload patterns shift. A ratio that was optimal last week may be wrong this week. We review the ratio every Monday based on the previous week's metrics.

# Prefill vs decode queue depth imbalance
avg(vllm_num_requests_waiting{pool="prefill"})
  / avg(vllm_num_requests_waiting{pool="decode"})

# Ratio > 2 sustained for 10 min = prefill bottleneck
# Ratio < 0.5 sustained for 10 min = decode bottleneck

Worked Example: Sizing a Production Cluster

Scenario: You need to serve 10,000 requests/hour with a 70B parameter model. Your SLO is TTFT p99 under 500ms. Average input length is 4K tokens. Average output length is 1K tokens. You are running on A100 80GB GPUs.

Step 1: Per-pod capacity estimate

• Model weights: 35 GB (FP16)
• Available for KV cache: 80 - 35 - 5 (overhead) = 40 GB
• KV cache per request at 4K input: ~256 MB
• Max concurrent requests per pod: 40000 / 256 = ~156
• But: TTFT p99 constraint limits effective concurrency. At 156 concurrent requests, queue depth causes TTFT to exceed 500ms. Practical limit: ~40 concurrent requests per pod to meet TTFT SLO.

Step 2: Throughput per pod

• Average request duration (prefill + decode): ~2.5 seconds
• At 40 concurrent requests with 2.5s average: 40 / 2.5 = 16 requests/second = 960 requests/minute = ~57,600 requests/hour per pod
• But this assumes 100% utilization. With queuing overhead and SLO constraints, practical throughput is ~60% of theoretical: ~34,500 requests/hour per pod

Step 3: Fleet size

• Base pods needed: 10,000 / 34,500 = 0.29 -- so 1 pod handles average load easily
• But: you need to handle bursts. If peak is 3x average (30,000 requests/hour): 30,000 / 34,500 = 0.87 -- still 1 pod, but with no headroom
• Add 40% headroom for spikes and maintenance: 0.87 * 1.4 = 1.2 -- round up to 2 pods
• Add 1 pod for rolling update capacity (maxSurge=1): 3 pods
• Minimum fleet size: 3 pods for this workload

Step 4: Cost estimate

• 3 A100 80GB GPUs, running 24/7
• At typical cloud pricing (~$2/GPU-hour): 3 * 24 * 30 = 2,160 GPU-hours/month = ~$4,320/month
• Cost per 1M tokens generated: $4,320 / (10,000 * 24 * 30 * 1000 / 1,000,000) = ~$0.60 per 1M output tokens

Rolling Updates and Zero-Downtime Deployments

Standard Kubernetes rolling update strategy was designed for applications that start in seconds. Inference pods that take 4-10 minutes to load a model break every assumption the rolling update controller makes about convergence time. We learned this during our second model update, when a "zero-downtime" rolling update caused 25 minutes of SLO violations because the default maxUnavailable=25% drained a quarter of our fleet while new pods sat loading weights.

Why Standard Rolling Updates Fail

Consider a 12-pod fleet with maxUnavailable=25% (the default). Kubernetes terminates 3 pods and starts 3 new ones. Those new pods take 8 minutes to become ready. During that window, you are serving all production traffic on 9 pods -- a 25% capacity reduction for 8 minutes. If load is near capacity, this triggers SLO violations immediately. And if any of those 9 remaining pods has an issue during the update, you are in trouble.

Staggered Rollout Strategy

# Rolling update config for GPU inference workloads
strategy:
  type: RollingUpdate
  rollingUpdate:
    maxSurge: 1        # Only add one pod at a time
    maxUnavailable: 0  # Never reduce current capacity
minReadySeconds: 120      # Verify stability before proceeding

With maxSurge=1 and maxUnavailable=0, the rollout creates one new pod, waits for it to become Ready (model fully loaded, warmup complete), serves traffic on it for minReadySeconds, and only then terminates one old pod. Capacity never drops below your pre-update fleet size. The trade-off: a 12-pod fleet takes 12 x (load time + minReadySeconds) to fully roll out. For an 8-minute model load plus 2-minute stability check, that is roughly 2 hours. This is the correct trade-off for production inference.

Model Pre-Caching with DaemonSets

The single most effective optimization for rollout speed is eliminating the model download from the critical path. Deploy a DaemonSet that keeps model weights on local NVMe storage. When a new model version is released, update the DaemonSet first and let it propagate across the fleet. By the time you start the inference deployment rollout, every node already has the model weights cached locally. Model load drops from 4-10 minutes to 45-90 seconds.

PodDisruptionBudgets for GPU Workloads

# PDB for inference workloads
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: inference-pdb
spec:
  minAvailable: "75%"  # Never drop below 75% capacity
  selector:
    matchLabels:
      app: llm-inference

Without a PDB, node maintenance, cluster autoscaler scale-downs, or even accidental kubectl operations can remove too many inference pods simultaneously. The PDB ensures that voluntary disruptions never reduce your serving capacity below a safe threshold.

Debugging Tools for Production Inference

When something goes wrong in production, you need to diagnose it fast. Standard Kubernetes debugging tools get you partway there, but inference workloads require additional GPU-specific and inference-specific diagnostic techniques. These are the commands we actually run during incidents, in roughly the order we run them.

GPU Diagnostics

# Check GPU utilization, memory, and temperature
nvidia-smi --query-gpu=index,name,utilization.gpu,memory.used,memory.total,temperature.gpu --format=csv

# Check GPU topology -- verify NUMA affinity (Scenario #2)
nvidia-smi topo -m

# Continuous GPU monitoring at 1-second intervals
nvidia-smi dmon -s pucvmet -d 1

# DCGM diagnostics for deeper GPU health checks
dcgmi diag -r 3  # Level 3 diagnostic (comprehensive)

vLLM Debug Endpoints

# Check model loaded status and server readiness
curl http://localhost:8000/health
curl http://localhost:8000/v1/models

# Get detailed server metrics (Prometheus format)
curl http://localhost:8000/metrics

# Check current running and waiting request counts
curl http://localhost:8000/metrics | grep vllm_num_requests

Network Debugging for Disaggregated Serving

# Verify RDMA device availability
ibstat
ibv_devinfo

# Test RDMA connectivity and bandwidth between pods
# On the server pod:
ib_write_bw -d mlx5_0 --report_gbits

# On the client pod:
ib_write_bw -d mlx5_0 --report_gbits <server-pod-ip>

# Check for RDMA errors and dropped packets
perfquery -x

Request Tracing Through the EPP

When a request takes an unexpected path or hits an unexpected latency, tracing its journey through the EPP reveals the routing decision logic:

• Check the EPP logs for the request ID to see which scoring factors influenced the routing decision
• Compare the pod selected vs. the pod that would have maximized cache hit probability
• Verify that the EPP had accurate state for all pods at the time of the routing decision (stale state from a failed health check can cause suboptimal routing)
• Check whether the request was rerouted due to a pod becoming unavailable during processing

Quick Diagnostic Runbook: TTFT Spike

When TTFT spikes, work through this in order. Stop at the first step that reveals the root cause.

1. Check queue depth across pods -- are requests backing up?
2. Check KV cache utilization -- are you hitting eviction pressure?
3. Check cache hit rate -- did it drop suddenly?
4. Check for pods in a loading or degraded state -- is the fleet capacity reduced?
5. Check for long-context requests that might be monopolizing prefill
6. Check GPU memory fragmentation -- can you still allocate contiguous blocks?
7. For disaggregated deployments, check RDMA throughput between pools

War Stories from Production

The scenarios above describe failure modes in isolation. In production, failures rarely arrive alone. These war stories illustrate how multiple failure modes compound, and how the resolution often requires looking beyond the immediate symptoms.

Deep Dives: 5 Incident Case Studies

The war stories above give you the narrative. These case studies give you the forensics. Each one includes the exact timestamps, the exact metrics, the exact commands we ran, and the exact resolution. Print these out and tape them to your monitor. You will need them.

# Check ECC error counts
nvidia-smi --query-gpu=ecc.errors.corrected.volatile.total,ecc.errors.uncorrected.volatile.total --format=csv
# Output: 1847, 1

# Check error history with DCGM
dcgmi diag -r 3 -j  # Level 3 diagnostic, JSON output

# Check if GPU should be retired
nvidia-smi --query-gpu=retired_pages.pending,retired_pages.sbe,retired_pages.dbe --format=csv
# Output: Yes, 12, 1  -- GPU has pages pending retirement

# The command that finally revealed the problem
openssl s_client -connect decode-pool-svc:8443 -servername decode-pool 2>&1 | grep -i "verify\|expire\|error"
# Output: Verify return code: 10 (certificate has expired)

# Check exact expiry
openssl s_client -connect decode-pool-svc:8443 2>/dev/null | openssl x509 -noout -dates
# Output: notAfter=Wed Jun 25 03:14:00 2026 GMT

kubectl get certificate -n inference -o wide
# Shows: READY=False, STATUS=Expired

# Force renewal
kubectl cert-manager renew --all -n inference

# Restart pods to pick up new certs
kubectl rollout restart deployment/prefill-pool -n inference
kubectl rollout restart deployment/decode-pool -n inference

# Diagnose CUDA kernel cache growth
python3 -c "
import torch
stats = torch.cuda.memory_stats()
print(f'Allocated: {stats[\"allocated_bytes.all.current\"] / 1e9:.3f} GB')
print(f'Reserved:  {stats[\"reserved_bytes.all.current\"] / 1e9:.3f} GB')
print(f'Active:    {stats[\"active_bytes.all.current\"] / 1e9:.3f} GB')
"

# Check CUDA context size
nvidia-smi --query-compute-apps=pid,used_memory --format=csv

kubectl rollout undo deployment/inference-server -n inference
# Rolling back to previous revision

# Flush KV cache on all serving pods
for pod in $(kubectl get pods -n inference -l app=inference-server -o name); do
  kubectl exec -n inference $pod -- curl -X POST http://localhost:8000/v1/cache/flush
done

# Verify cache is empty
kubectl exec -n inference deploy/inference-server -- curl -s http://localhost:8000/metrics | grep cache_usage

Severity Classification Guide

Every incident needs a consistent severity classification to drive the right response. This guide defines the severity levels we use for LLM inference incidents, calibrated to the failure modes described throughout this post. Use it to set expectations with your on-call rotation and stakeholders.

How llm-d's EPP Prevents Production Issues

The Endpoint Picker Policy is the control plane for inference routing in llm-d. Rather than distributing requests blindly with round-robin or random selection, the EPP observes the real-time state of every serving pod and makes informed routing decisions. It is the single component that ties together the lessons from every scenario described in this post.

The EPP continuously monitors queue depth, KV cache contents, GPU memory pressure, model loaded status, and network health across the entire serving fleet. Every routing decision factors in this state, which is why many of the incidents described earlier are preventable -- or at least significantly mitigated -- with EPP-informed routing.

• Cache-aware routing -- Routes requests to pods that already have relevant prefix cache entries. When a request arrives with a system prompt that matches cached prefixes on a specific pod, the EPP sends it there. This can reduce TTFT from hundreds of milliseconds to tens of milliseconds for cache-hit requests. This directly prevents the cache eviction storms described in Scenario #1 by concentrating cache usage rather than scattering it.
• Load-aware routing -- Avoids sending requests to pods that are already saturated. Instead of blindly distributing load, the EPP factors in each pod's current queue depth and GPU utilization, preventing the hot-spotting that causes the latency spikes in Scenario #11. Requests are distributed based on actual capacity, not assumed capacity.
• Health-aware routing -- Automatically routes around pods that are loading models, running defragmentation, or experiencing issues. No traffic reaches a pod that cannot serve it, eliminating the "healthy but not ready" failure mode from Scenario #6 entirely. This also handles the defrag-aware routing needed for Scenario #10.
• Session affinity -- Keeps multi-turn conversations on the same pod to maximize KV cache reuse. The second message in a conversation benefits from the cached context of the first, dramatically improving TTFT for conversational workloads. This also helps with tenant isolation (Scenario #4) by keeping tenant traffic on consistent pods.
• Context-length-aware scheduling -- The EPP can differentiate between short-context and long-context requests, routing them to appropriate priority queues or dedicated pod pools. This prevents the prefill starvation described in Scenario #11 where 128K requests monopolize compute that short requests need.
• Disaggregation-aware routing -- For disaggregated deployments, the EPP factors in the health and performance of cross-pool communication paths. If RDMA throughput degrades (Scenario #7), the EPP can dynamically adjust routing to compensate -- including falling back to co-located serving when the disaggregated path is unreliable.

The critical insight is that most production inference incidents are not individual component failures -- they are system-level emergent behaviors that arise from the interaction between routing, caching, memory management, and load. The EPP is the point where all of these concerns converge, and it is the right place to implement the cross-cutting logic that prevents cascading failures.

For architecture details, configuration reference, and contribution guidelines, see the llm-d documentation and source.

Your First Week Deploying llm-d in Production

You do not need to solve every problem in this post on day one. But there is a minimum viable set of operational practices that will keep you out of the worst trouble. This is the checklist we wish we had when we started. Do these in your first week and you will avoid the most common early incidents.

Security Considerations for Production LLM Inference

Security in LLM inference is not optional, and it is not something you bolt on after launch. We learned several security lessons the hard way, and a few we were fortunate enough to learn from other teams' incidents before they became ours.

Model Endpoint Authentication

Every inference endpoint must require authentication. This sounds obvious, but in the rush to get models serving, it is remarkably common to deploy with authentication disabled "temporarily" and forget to enable it. We caught an internal team running an unauthenticated inference endpoint on a routable IP for 6 days.

• mTLS between all inference components: Prefill pods, decode pods, EPP, and the API gateway should all communicate over mTLS. Use cert-manager for automated certificate lifecycle (see Case Study 2 for what happens when you do not).
• API key rotation: Rotate API keys every 90 days minimum. Use Kubernetes Secrets with external secret stores (HashiCorp Vault, AWS Secrets Manager) -- never store keys in ConfigMaps.
• JWT token validation: For user-facing endpoints, validate JWT tokens at the gateway layer before requests reach the inference pods. This prevents unauthenticated requests from consuming GPU compute.

Network Segmentation

Inference pods should not have outbound internet access. Period. A compromised model serving process with internet access is a data exfiltration vector. Use Kubernetes NetworkPolicies to restrict traffic flow:

# NetworkPolicy: inference pods can only talk to each other and the API gateway
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: inference-network-policy
  namespace: inference
spec:
  podSelector:
    matchLabels:
      app: llm-inference
  policyTypes:
    - Ingress
    - Egress
  ingress:
    - from:
      - namespaceSelector:
          matchLabels:
            name: api-gateway
      - podSelector:
          matchLabels:
            app: llm-inference
  egress:
    - to:
      - podSelector:
          matchLabels:
            app: llm-inference
    - to:  # Allow DNS resolution
      - namespaceSelector: {}
        podSelector:
          matchLabels:
            k8s-app: kube-dns
      ports:
        - port: 53
          protocol: UDP

Prompt Injection Defense

Prompt injection is real and it targets your inference layer. Defense-in-depth applies here:

• Input validation: Reject requests with known injection patterns before they reach the model. This is not foolproof but catches the obvious attacks.
• Output filtering: Scan model outputs for sensitive patterns (API keys, internal URLs, PII) before returning to the client. Log filtered responses for review.
• System prompt isolation: System prompts should be injected server-side, never accepted from client input. The client should not be able to override or view the system prompt.
• Rate limiting by token count: Limit not just requests per second but tokens per minute per client. A prompt injection attack that generates 100K output tokens costs you GPU time even if it does not succeed in extracting data.

Audit Logging

Log every inference request with: timestamp, client ID, model version, input token count, output token count, finish_reason, and latency. Do not log the actual prompt or response content in the primary audit log -- that creates a PII liability. If you need content logging for debugging, use a separate encrypted store with strict access controls and automatic 30-day retention.

Model Integrity Verification

Model weights are code. Treat them with the same supply chain security you apply to container images.

• Verify model checksums on load: Every model artifact has a SHA-256 hash computed at build time. The model loading process verifies this hash before loading weights onto the GPU. A hash mismatch aborts the load and alerts the security team.
• Sign model artifacts: Use cosign or Notation to sign model files. The DaemonSet that pre-pulls model weights verifies signatures before writing to local NVMe. Unsigned artifacts are rejected.
• Audit the model supply chain: Know where your model came from. Track the training data provenance, fine-tuning history, and every transformation applied. If you are pulling models from a public registry, verify the publisher. An attacker who can replace your model weights controls your model's output.

Secret Management

• Kubernetes Secrets with encryption at rest: The minimum viable approach. Enable EncryptionConfiguration in the API server with an AES-256 key. This encrypts Secret data in etcd. Without this, Secrets are base64-encoded (not encrypted) and readable by anyone with etcd access.
• External secret managers: For production, use Vault, AWS Secrets Manager, or Azure Key Vault with the External Secrets Operator. Secrets are fetched at pod startup and injected as environment variables or mounted files. They never persist in the Kubernetes API.
• Rotate secrets on a schedule: Model registry tokens, API keys, database credentials -- all should rotate automatically. Never assume a secret is safe just because it has not been compromised yet.

RBAC for Inference Namespaces

# RBAC: restrict who can modify inference deployments
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: inference-operator
  namespace: inference
rules:
  - apiGroups: [""]
    resources: ["pods", "services", "configmaps"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["apps"]
    resources: ["deployments"]
    verbs: ["get", "list", "watch", "update", "patch"]
  - apiGroups: ["apps"]
    resources: ["deployments/rollback"]
    verbs: ["create"]  # Allow rollback but not delete

What's Next

Production LLM inference is still a young discipline. The problems described in this post are actively being worked on by the llm-d community and by teams across the industry. Several areas of active development deserve attention:

• Predictive autoscaling -- Moving beyond reactive HPA to scaling decisions based on historical patterns, calendar awareness, and real-time demand signals. The goal is to have new pods loaded and ready before the traffic spike arrives.
• Smarter KV cache management -- Better eviction policies, cross-pod cache sharing, and hierarchical caching (GPU HBM, CPU DRAM, NVMe) to extend effective cache capacity without additional GPU memory.
• Fragmentation-aware memory allocation -- Memory allocators designed specifically for the allocation patterns of inference workloads, reducing fragmentation without requiring defragmentation cycles.
• Automated capacity planning -- Tools that analyze your workload distribution and SLO requirements to recommend fleet size, disaggregation ratios, and headroom budgets.
• Improved observability tooling -- Standardized metrics, pre-built Grafana dashboards, and inference-specific alerting rules that work out of the box so teams do not have to build monitoring from scratch.

If your team has production inference experience, the llm-d community would benefit from it. The project is a CNCF Sandbox project, and contributions are welcome -- whether that is code, documentation, operational patterns from your own environment, or simply sharing the failure modes you have encountered so others can learn from them.

The hardest part of production inference is the knowledge that only comes from operating it under real conditions. Every team that shares their operational experience makes the entire ecosystem more resilient. That is what this post aims to do, and we hope it helps you avoid at least a few of the 3 AM pages that taught us these lessons.

Pre-Production Review Checklist

Before your inference service goes live, walk through this checklist with your team. Each item maps to a failure mode covered in this post. Completing all of them will not guarantee zero incidents, but it will eliminate the most common ones.

Further Reading

These resources provide deeper coverage of the tools, platforms, and techniques referenced throughout this post.