llm-d Monitoring & Observability on OpenShift

You Can’t Optimize What You Can’t Measure

LLM inference looks simple from the outside: prompt in, response out. But underneath, there’s a GPU memory allocator managing KV cache blocks, a routing layer scoring pods by prefix match quality, and a token generation pipeline sensitive to batch sizes, queue depths, and thermal throttling.

When latency spikes from 200ms to 4 seconds, you need to know which layer is the bottleneck -- not just that something is slow. That’s what llm-d monitoring gives you.

The Three Metric Layers

Layer 1: EPP routing metrics (~50 metrics)

The Endpoint Picker Policy exposes metrics on port 9090 via a ServiceMonitor. These tell you how routing decisions are being made:

• Prefix cache hit rate -- what percentage of requests found a cached prefix match. Target: >80%
• Pool saturation -- EPP flow control signal. 0.3–0.7 is healthy; >1.0 means the EPP is throttling requests
• Prefix indexer hit ratio -- median quality of prefix matches across the fleet

Layer 2: vLLM engine metrics (~386 metrics)

Each vLLM replica exposes metrics on port 8000 via a PodMonitor. These show what’s happening inside the inference engine:

• TTFT p95 -- time-to-first-token at the 95th percentile. Healthy: <200ms with cache hits
• Token throughput -- generation tokens per second across all pods
• KV cache utilization -- fraction of cache blocks in use. >90% triggers LRU eviction; >95% degrades hit rate

Layer 3: GPU infrastructure (DCGM metrics)

NVIDIA’s Data Center GPU Manager provides hardware-level telemetry via the GPU Operator:

• GPU utilization -- compute percentage
• Framebuffer memory -- GPU memory consumption
• Temperature -- thermal state (throttling starts at 83°C on most GPUs)

Setting It Up on OpenShift

llm-d monitoring runs on OpenShift’s built-in monitoring stack -- Prometheus and Grafana, no external tools needed.

# Enable user workload monitoring
oc apply -f - <<EOF
apiVersion: v1
kind: ConfigMap
metadata:
  name: user-workload-monitoring-config
  namespace: openshift-user-workload-monitoring
data:
  config.yaml: |
    prometheus:
      retention: 24h
      resources:
        requests:
          cpu: 200m
          memory: 1Gi
EOF

Then deploy the ServiceMonitor for EPP metrics and PodMonitor for vLLM metrics. The companion web app includes the full YAML configs and a real dashboard.json you can import directly into Grafana.

What to Watch: Five Failure Scenarios

1. Cache-miss storm

Prefix cache hit rate drops below 50%. TTFT spikes as every request does full prefill. Cause: traffic pattern changed, or KV cache is too small for the working set.

2. Pod restart recovery

A pod restarts and loses its KV cache. Hit rate dips temporarily as the cache warms back up. Watch for: cascading restarts that keep the cache cold.

3. Traffic spike

Pool saturation exceeds 1.0. EPP throttles incoming requests. Queue depth grows. Action: scale replicas or shed lower-priority traffic.

4. KV cache pressure

Utilization exceeds 92%. LRU eviction starts dropping cached prefixes. Hit rate degrades. Action: increase KV cache allocation or reduce max sequence length.

5. GPU thermal throttling

Temperature exceeds 85°C. GPU clock speed drops. Throughput decreases 15–30%. Action: check cooling, reduce batch size, or redistribute load.

Six Alerts You Should Set

Try It

The interactive monitoring simulator lets you explore all five failure scenarios without a cluster. Switch between healthy and degraded states, watch metrics respond in real time, and copy the PromQL queries and alert rules for your own deployment.

The dashboard.json ships in the GitHub repo -- import it into Grafana and you’re monitoring llm-d in production.

• llm-d on GitHub: github.com/llm-d/llm-d
• Community: #llm-d on CNCF Slack