LLM inference observability with llm-d

01 Why LLM inference needs its own observability

If you've run a Kubernetes cluster for any length of time, you know the standard monitoring playbook: CPU, memory, pod restarts, network throughput, HTTP error rates. That works great for normal microservices. A request comes in, burns some CPU and memory, touches a database, and returns in a few milliseconds. When something breaks, CPU spikes or memory leaks or error rates climb, and your dashboards point right at it. That playbook is old, well-worn, proven — and catastrophically insufficient for LLM inference.

The core problem: standard Kubernetes monitoring was designed for stateless request-response services, and LLM inference is a different computational model entirely. A single request can eat gigabytes of GPU memory, run for seconds or minutes, and juggle internal state that has no equivalent in a normal web service. The failure modes are alien if you're coming from cloud-native ops, and the metrics that matter are ones Kubernetes doesn't even know exist.

Take the most deceptive metric in GPU computing: utilization. Your DCGM exporter reports 92% GPU utilization across all pods, and your instinct says everything's running smoothly. But behind that number, every single request might be doing cold prefill because your KV cache hit rate collapsed. Users are seeing five-second time-to-first-token while your dashboards are green across the board. The GPU is busy. It's just busy doing redundant work that a working cache would've skipped entirely.

So what do these alien failure modes look like? Start with KV cache eviction storms: memory pressure forces cached key-value pairs out, so requests that would've been cache hits become cold prefills, which burns more GPU compute, which raises memory pressure further, which triggers more evictions. That feedback loop can take a healthy cluster from sub-100ms TTFT to multi-second TTFT in under a minute, and standard monitoring won't register anything unusual until users start filing complaints.

Adapter swap thrashing is another one. If you serve multiple LoRA adapters on the same base model, the system loads and unloads adapter weights as requests arrive for different fine-tuned variants. If your routing isn't adapter-aware, you can end up with every request triggering a swap, turning what should be a few milliseconds of overhead into a constant drag on throughput. Prefill/decode imbalance is yet another invisible killer: too many requests in the compute-heavy prefill phase at once and decode throughput for in-flight requests collapses, because prefill hogs a disproportionate share of GPU compute and memory bandwidth. And then there's head-of-line blocking. vLLM uses continuous batching to process multiple requests at once, but a single request with a massive context window can monopolize the attention computation and starve shorter requests of GPU cycles. Without metrics that separate prefill from decode, cache hits from misses, adapter loads from steady-state inference, you're flying blind in a system that fails in ways your existing monitoring was never built to see.

02 The metrics that actually matter

Monitoring LLM inference means tracking a different set of metrics — ones that capture autoregressive token generation, KV cache behavior, and GPU memory management. Honestly, six core metrics tell you almost everything you need to know about system health and user experience.

Time-to-First-Token (TTFT)

TTFT is the time from a request arriving at the inference server to the first token of the response. It's basically your prefill latency, which is the dominant cost for long-context requests. It's also the metric users feel most: the pause between pressing send and seeing the model start to respond — in a streaming UI, the silence before the first word. Track it at multiple percentiles: p50 for the typical experience, p95 for the tail that hits one in twenty users, p99 for the worst-case behavior that drives complaints and SLA violations. A healthy cluster serving Llama 3.1 70B should target p50 TTFT under 500ms and p99 under 3 seconds for typical prompt lengths. If p99 goes over 5 seconds, something's wrong. Over 10 seconds, your users are actively churning. A cache hit can cut TTFT from seconds to milliseconds, which is why cache-aware routing is the single most impactful optimization in the whole stack.

Inter-Token Latency (ITL)

ITL is the gap between consecutive tokens during decode. TTFT is the initial wait; ITL is the perceived streaming speed. Users are surprisingly sensitive to ITL variation — a consistent 30ms feels smooth and natural, while ITL oscillating between 10ms and 200ms feels stuttery and broken even if the average is identical. ITL spikes usually mean decode contention: new prefills getting scheduled into the continuous batch, GPU memory bandwidth saturation, or thermal throttling. Track p50 and p99, and alert when p99 ITL goes over 100ms. Worth keeping separate from TTFT because the root causes and fixes differ: TTFT problems are usually cache or queue related, ITL problems are usually contention or hardware related.

Throughput: Tokens per Second

Split throughput into prefill and decode — they compete for different GPU resources and scale differently. Prefill is compute-bound and benefits from tensor parallelism and larger batches. Decode is memory-bandwidth-bound and also likes higher batch sizes, but it's constrained by the autoregressive nature of generation. Aggregate tokens-per-second is fine for capacity planning and cost math, but the prefill/decode split is what you need for diagnosis. Decode throughput suddenly dropping while prefill holds constant is a strong signal that too many requests are entering prefill at once and starving the decode pipeline.

KV Cache Hit Ratio

This one is arguably the single most important metric for an llm-d cluster. It's the fraction of incoming request tokens served from cached key-value pairs instead of computed from scratch during prefill. In a well-tuned cluster with cache-aware routing, you should see hit ratios above 60% for workloads with prompt reuse — chatbots with system prompts, RAG apps with repeated document contexts, code completion with shared file contexts. Below 40% means your routing layer isn't steering requests to pods that already cached the prefix, and you're paying a huge latency and compute tax for redundant prefill. When that happens, check whether the EPP cache affinity weight is too low, whether your cache is too small and evicting prematurely, or whether your traffic genuinely has no prefix overlap.

Queue Depth and GPU Memory Pressure

Queue depth is how many requests are waiting at each inference pod. With continuous batching, a small non-zero queue is normal and healthy, but persistent depths above 20-30 mean the cluster is under-provisioned or a few pods are hoarding traffic from routing imbalance. Watch for asymmetry: one pod with a queue of 60 while others sit at 2 means your load balancing is misconfigured. And track GPU memory utilization separately from GPU compute utilization — different failure thresholds, different fixes. Memory pressure above 90% triggers KV cache evictions, which tanks hit ratios, which raises compute demand, which raises memory pressure further. That's the eviction storm feedback loop, and catching it early means alerting on GPU memory well before it hits the point where evictions start cascading.

03 Building the monitoring stack

The foundation is Prometheus for metrics and Grafana for visualization. vLLM exposes a metrics endpoint at /metrics with Prometheus-formatted metrics covering request latencies, token throughput, cache statistics, and internal queue state. The architecture is simple: each vLLM pod exposes metrics on a configurable port, Prometheus scrapes on an interval, and Grafana dashboards query Prometheus for real-time and historical views. You don't need to instrument anything yourself — vLLM ships all of this out of the box.

First step: get Prometheus discovering and scraping your vLLM pods. If you're on Kubernetes with the Prometheus Operator, the cleanest way is a ServiceMonitor resource that discovers pods by label selector. No manually maintained scrape target lists as pods scale up and down — which matters a lot when horizontal pod autoscaling is active.

# ServiceMonitor for vLLM inference pods
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: vllm-inference-monitor
  namespace: llm-d
  labels:
    release: prometheus
spec:
  selector:
    matchLabels:
      app: vllm-inference
  endpoints:
    - port: metrics
      interval: 15s
      path: /metrics
      scrapeTimeout: 10s
  namespaceSelector:
    matchNames:
      - llm-d

Not on the Prometheus Operator? Configure static scrape targets or use Kubernetes service discovery directly in your Prometheus config. The two knobs that matter: scrape interval, which should be 15 seconds for production LLM workloads so you catch fast-moving cache state and queue depth, and scrape timeout, which should be 10 seconds so healthy-but-busy pods don't get marked unresponsive under heavy load.

# prometheus.yml scrape config for vLLM pods
scrape_configs:
  - job_name: 'vllm-inference'
    kubernetes_sd_configs:
      - role: pod
        namespaces:
          names:
            - llm-d
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_label_app]
        regex: vllm-inference
        action: keep
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_port]
        action: replace
        target_label: __address__
        regex: (.+)
        replacement: ${1}:8000
      - source_labels: [__meta_kubernetes_pod_name]
        target_label: pod
      - source_labels: [__meta_kubernetes_pod_label_model]
        target_label: model

Dashboard organization matters more than people realize. The common mistake is one monolithic dashboard with fifty panels that nobody can parse during an incident. Instead, build a hierarchy: a top-level overview for cluster-wide health at a glance, a pod-level dashboard for individual inference servers, a cache dashboard for KV cache behavior and routing effectiveness, and a cost dashboard for GPU utilization efficiency and cost-per-token. Each one answers a specific question. The overview answers "is the cluster healthy right now?" The pod dashboard answers "why is this specific pod underperforming?" The cache dashboard answers "is the routing layer doing its job?" That separation keeps each dashboard scannable instead of hitting a stressed operator with an undifferentiated wall of charts.

04 The observability tax

Monitoring isn't free. Every scrape, every trace, every log line burns compute, network, and storage that could otherwise be serving inference. The goal is maximum visibility with minimum impact, and getting there takes deliberate choices about scrape intervals, sampling rates, polling frequencies, and log volume.

Prometheus scrapes cost CPU on the inference pods in a way that's easy to overlook. Each scrape makes the vLLM metrics endpoint serialize hundreds of counters into the Prometheus text format, and on a busy pod that serialization competes with the inference engine for CPU cycles. At a 15-second interval the overhead is negligible for most deployments. At 5 seconds across 20+ pods it becomes measurable — especially on pods where CPU is already tight, because most of the compute happens on the GPU and the CPU allocation is minimal. The serialization itself is fast, but vLLM exposes thousands of histogram buckets for the TTFT and ITL distributions, and iterating over all of them and formatting text adds up across frequent scrapes.

Trace sampling is a straight tradeoff between visibility and storage. 100% sampling doesn't work at production scale — a cluster at 1,000 requests per second generates millions of spans per hour, which means enormous storage and slow queries in Jaeger or Tempo. Tail-based sampling is the move: keep 100% of traces that exceed a latency threshold (say, TTFT > 3 seconds) and a small slice (5-10%) of normal traces for baseline visibility. You capture every problematic request while keeping storage sane. You never miss the traces that matter most — the ones where something went wrong — and you still hold a representative sample of healthy requests for comparison and trend analysis.

The DCGM exporter has its own overhead. It polls GPU counters at a configurable frequency, and 1-second polling gives you granular visibility but puts measurable load on the GPU driver — microseconds of jitter on CUDA kernel launches, which mostly doesn't matter but can compound during latency-sensitive decode iterations where consistent timing counts. For most production deployments, polling every 5-10 seconds is plenty for alerting and trends without meaningful performance impact. Save 1-second polling for targeted debugging: bump the frequency on the affected node, grab the data you need, then put the production interval back.

Log volume is the other tax, and it scales linearly with traffic. Structured JSON logging from vLLM, the EPP, and the Gateway adds up fast: a single request can produce 5-10 log lines across the stack, and at 1,000 requests per second that's 5,000 to 10,000 lines per second through your pipeline. Size your aggregation (Loki, Elasticsearch, or CloudWatch) accordingly, and filter log levels at the source rather than at the aggregator to save network bandwidth. Setting vLLM to WARNING in production kills the per-request INFO lines that dominate volume while keeping the errors and warnings you actually need for diagnosis.

Concrete numbers, because "monitor wisely" is useless advice: use 15-second Prometheus scrape intervals — the sweet spot between granularity and overhead for LLM inference. Combine 5-10% head-based sampling with 100% tail-based sampling for traces that exceed your SLA thresholds. Set DCGM polling to 5 seconds in production. Keep high-cardinality metrics for 7 days, which covers most incident investigations, and aggregated rollups for 90 days for capacity planning. None of these numbers are arbitrary — they come from running LLM inference clusters where the cost of over-instrumenting shows up as GPU cycles that could have been tokens.

05 PromQL queries for the core dashboards

PromQL is where Prometheus earns its keep. For LLM inference you need queries that go past simple rate calculations to capture autoregressive generation, cache dynamics, and multi-model serving. These are the ones on every llm-d dashboard I build, with what each measures and how to use it in Grafana panels.

TTFT Percentiles

The most important query in the whole stack: time-to-first-token at the 99th percentile. This one number tells you more about user experience than any other metric you have.

# P99 Time-to-First-Token
histogram_quantile(0.99,
  rate(vllm_request_ttft_seconds_bucket[5m])
)

# P95 and P50 for comparison
histogram_quantile(0.95, rate(vllm_request_ttft_seconds_bucket[5m]))
histogram_quantile(0.50, rate(vllm_request_ttft_seconds_bucket[5m]))

Put p50, p95, and p99 on the same Grafana panel as overlaid time series. When p99 diverges hard from p50, most requests are fast but a subset is hitting a pathological code path — typically cold prefill on cache misses or long queue waits from contention.

Token Throughput

# Aggregate token generation rate across all pods
sum(rate(vllm_tokens_generated_total[1m]))

# Per-pod token throughput for identifying stragglers
rate(vllm_tokens_generated_total[1m])

Chart the per-pod query as a stacked area to see how throughput is distributed. If one pod consistently generates fewer tokens than its peers, suspect a hardware issue, a cache problem, or a routing imbalance sending its traffic elsewhere.

KV Cache Hit Rate

# Cache hit ratio (percentage of prefix tokens served from cache)
sum(rate(vllm_cache_hits_total[5m])) /
(sum(rate(vllm_cache_hits_total[5m])) + sum(rate(vllm_cache_misses_total[5m])))
* 100

Format this as a percentage with color thresholds: green above 60%, yellow between 40% and 60%, red below 40%. This one panel tells you whether the EPP's cache-aware routing is actually earning its keep or whether requests are getting scattered across pods with no regard for cache affinity.

GPU Memory, Queue Depth, Request Rate, and Error Rate

# GPU memory utilization per pod (via DCGM exporter)
DCGM_FI_DEV_FB_USED / DCGM_FI_DEV_FB_TOTAL * 100

# Queue depth across all pods
sum(vllm_num_requests_waiting)

# Request rate by model (for multi-model deployments)
sum by (model) (rate(vllm_request_total[5m]))

# Error rate (5xx and OOM)
sum(rate(vllm_request_errors_total[5m])) /
sum(rate(vllm_request_total[5m])) * 100

These queries are the backbone of the operational dashboards. Each answers one question: are users waiting too long? Are we generating tokens fast enough? Is the cache working? Are GPUs running out of memory? Are requests piling up? Which models get the most traffic? Are we throwing errors? Wire them into Grafana panels with color-coded thresholds and you've got a stack that shows the real health of your cluster instead of the misleading picture generic Kubernetes metrics paint.

06 The 10 PromQL queries every llm-d operator needs

These are the ten queries to keep bookmarked, pinned, and ready to paste into any Grafana panel. Each targets one operational question. Together they cover latency, throughput, caching, cost, and hardware health for an llm-d cluster.

1. P99 Time-to-First-Token
   The single most important user-experience metric. Alerts you to cache misses, queue buildup, or prefill contention before users complain.
   PromQL Copy

   histogram_quantile(0.99,
     rate(vllm_request_ttft_seconds_bucket[5m])
   )

2. Cluster Token Throughput (tokens/sec)
   Aggregate decode throughput across all pods. A sustained drop means you are losing capacity -- correlate with GPU memory and queue depth.
   PromQL Copy

   sum(rate(vllm_tokens_generated_total[1m]))

3. KV Cache Hit Rate (%)
   Percentage of prefix tokens served from cache. Below 40% means your routing is not exploiting prefix reuse and you are paying a large compute tax.
   PromQL Copy

   sum(rate(vllm_cache_hits_total[5m])) /
   (sum(rate(vllm_cache_hits_total[5m])) +
    sum(rate(vllm_cache_misses_total[5m])))
   * 100

4. GPU Memory Utilization (%)
   Per-GPU framebuffer usage. Above 90% you are in the danger zone for KV cache eviction storms. Alert well before you get there.
   PromQL Copy

   DCGM_FI_DEV_FB_USED / DCGM_FI_DEV_FB_TOTAL * 100

5. Total Queue Depth
   Requests waiting across the cluster. Sustained values above 50 mean you need more replicas or your routing is creating hotspots.
   PromQL Copy

   sum(vllm_num_requests_waiting)

6. Request Error Rate (%)
   Percentage of requests returning errors. Above 1% sustained is a warning. Above 5% is an active incident.
   PromQL Copy

   sum(rate(vllm_request_errors_total[5m])) /
   sum(rate(vllm_request_total[5m])) * 100

7. Adapter Swap Rate (swaps/sec)
   How frequently LoRA adapters are being loaded and unloaded. A high swap rate means adapter-aware routing is not grouping requests by adapter, and every swap adds latency.
   PromQL Copy

   sum(rate(vllm_lora_adapter_swaps_total[5m]))

8. Prefill / Decode Ratio
   Ratio of requests currently in the prefill phase vs. decode phase. A ratio above 0.5 means more than a third of in-flight requests are doing prefill, which starves decode throughput and causes ITL spikes.
   PromQL Copy

   sum(vllm_num_requests_prefill) /
   clamp_min(sum(vllm_num_requests_decode), 1)

9. Cost per Million Tokens (USD)
   Real-time unit economics. Divides your GPU fleet cost (derived from GPU count and hourly rate) by actual token throughput. Track this daily to catch efficiency regressions.
   PromQL Copy

   # Assumes $35/hr per GPU; adjust the constant for your rate
   # count(DCGM_FI_DEV_GPU_TEMP) returns number of GPUs in the fleet
   (
     count(DCGM_FI_DEV_GPU_TEMP) * 35 / 3600   # fleet cost per second
   )
   /
   clamp_min(sum(rate(vllm_tokens_generated_total[5m])), 0.001)
   * 1e6

10. Cache Eviction Rate (evictions/sec)
    Rate at which KV cache entries are being evicted. A spike here is the leading indicator of an eviction storm -- it precedes TTFT degradation by 30-60 seconds, giving you a narrow window to act.
    PromQL Copy

    sum(rate(vllm_cache_evictions_total[5m]))

07 Advanced PromQL queries

Past the essential ten, you'll eventually want deeper visibility into cache behavior, adapter management, routing efficiency, and hardware health. These eight queries cover that next level. Each targets a specific blind spot the essential queries don't reach.

1. KV Cache Fragmentation Rate
   Ratio of allocated-but-wasted cache blocks to total allocated blocks. High fragmentation means the cache is reserving memory that is not being used effectively, reducing the usable cache capacity. Sustained fragmentation above 20% indicates that the block allocator needs tuning or that request patterns are causing excessive partial block allocation.
   PromQL Copy

   (vllm_cache_blocks_allocated - vllm_cache_blocks_used) /
   clamp_min(vllm_cache_blocks_allocated, 1) * 100

2. Prefix Sharing Efficiency Ratio
   Measures how effectively requests share cached prefixes across the cluster. A high ratio means the EPP is successfully routing requests with common prefixes to the same pods, maximizing cache reuse. Below 30% means prefix-aware routing is not delivering meaningful benefits and you should review EPP cache affinity weights.
   PromQL Copy

   sum(rate(vllm_prefix_cache_shared_tokens_total[5m])) /
   clamp_min(sum(rate(vllm_prefix_cache_total_tokens_total[5m])), 1)
   * 100

3. LoRA Adapter Switch Frequency per Pod
   Identifies pods suffering from adapter thrashing, where adapters are loaded and unloaded so frequently that swap overhead dominates useful inference time. Pods with a swap rate above 2 per minute are likely experiencing latency degradation from constant weight loading. Group by pod to identify which specific pods need routing adjustments.
   PromQL Copy

   sum by (pod) (rate(vllm_lora_adapter_swaps_total[5m])) * 60

4. EPP Routing Decision Distribution
   Shows whether traffic is balanced across pods or whether the EPP is creating hotspots by repeatedly selecting the same pods. A healthy distribution has a coefficient of variation below 0.3 across pods. Values above 0.5 indicate significant routing imbalance that warrants investigation of scoring weights.
   PromQL Copy

   stddev(sum by (pod) (rate(vllm_request_total[5m]))) /
   clamp_min(avg(sum by (pod) (rate(vllm_request_total[5m]))), 0.001)

5. Token Generation Throughput by Adapter
   Per-adapter performance comparison that reveals whether certain LoRA adapters are generating tokens significantly slower than others. Adapter-specific performance differences can indicate that certain adapter weights interact poorly with the base model's KV cache patterns, or that adapter size is causing memory pressure on specific pods.
   PromQL Copy

   sum by (adapter) (rate(vllm_tokens_generated_total[5m]))

6. Queue Wait Time Histogram
   Distribution of how long requests wait in the inference queue before starting prefill. The p99 of queue wait time is a leading indicator of capacity problems: it rises before TTFT degrades because queue time is a component of TTFT. Alert on p99 queue wait above 2 seconds.
   PromQL Copy

   histogram_quantile(0.99,
     rate(vllm_request_queue_time_seconds_bucket[5m])
   )

7. GPU Thermal Headroom
   How close GPUs are to thermal throttling thresholds. Most datacenter GPUs begin throttling at 83-85 degrees Celsius. Tracking thermal headroom (the gap between current temperature and throttle point) provides early warning of cooling failures or workload-driven thermal issues. Below 10 degrees of headroom, performance may already be degrading due to dynamic frequency scaling.
   PromQL Copy

   85 - DCGM_FI_DEV_GPU_TEMP

8. Inter-Token Latency Stability (ITL Coefficient of Variation)
   Variance in ITL that indicates decode contention. A low coefficient of variation (below 0.3) means consistent, smooth token generation. Values above 0.5 indicate that the decode phase is experiencing significant jitter, usually caused by competing prefill operations, memory bandwidth saturation, or thermal throttling interfering with decode iterations.
   PromQL Copy

   stddev_over_time(
     histogram_quantile(0.50, rate(vllm_request_itl_seconds_bucket[1m]))[10m:]
   )
   /
   clamp_min(
     avg_over_time(
       histogram_quantile(0.50, rate(vllm_request_itl_seconds_bucket[1m]))[10m:]
     ), 0.0001
   )

08 Alerting strategy

Not everything that belongs on a dashboard deserves an alert. Alert fatigue is the fastest way to get real incidents ignored, because operators get conditioned to dismiss notifications as noise. The rule is simple: dashboards are for exploration and context during investigation, alerts are for situations that need a human within a defined time window. If a metric going out of range doesn't require someone to act, it's a dashboard panel, not an alert rule.

For LLM inference there's a clear hierarchy of alertable conditions. The thresholds below are calibrated for llm-d deployments serving 70B-parameter models on A100 or H100 GPUs — adjust for your model size, hardware generation, and SLAs.

Warning-level alerts

• TTFT p99 above 5 seconds sustained for 5 minutes. This indicates emerging latency problems, typically declining cache hit rates or increasing queue depth. Investigate promptly but do not necessarily page anyone at 3am.
• KV cache hit ratio below 40% sustained for 10 minutes. Your routing layer may not be directing requests to pods with cached prefixes effectively. Check EPP scoring weights and pod traffic distribution.
• GPU memory utilization above 90% sustained for 5 minutes. You are approaching the threshold where KV cache evictions begin cascading. Consider scaling horizontally or tuning your cache eviction policy before the situation degrades further.
• Error rate above 1% sustained for 5 minutes. Occasional errors are expected in any distributed system, but sustained errors above one percent indicate a systemic issue that requires investigation.

Critical-level alerts

• TTFT p99 above 10 seconds sustained for 3 minutes. Users are having a terrible experience and this is an active incident requiring immediate response.
• Queue depth above 50 sustained for 5 minutes. The cluster cannot keep up with incoming request volume. Scale immediately or activate load shedding.
• Pod restarts above 3 within a 15-minute window. Something is causing pods to crash-loop, most likely OOM kills from GPU memory exhaustion or a misconfigured model configuration.

# PrometheusRule for llm-d alerting
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: llm-d-inference-alerts
  namespace: llm-d
spec:
  groups:
    - name: llm-d.latency
      rules:
        - alert: HighTTFTWarning
          expr: |
            histogram_quantile(0.99,
              rate(vllm_request_ttft_seconds_bucket[5m])
            ) > 5
          for: 5m
          labels:
            severity: warning
          annotations:
            summary: "TTFT p99 exceeds 5s"
            description: "P99 time-to-first-token is {{ $value | humanizeDuration }}"
            runbook_url: "https://runbooks.example.com/llm-d/HighTTFT"
        - alert: HighTTFTCritical
          expr: |
            histogram_quantile(0.99,
              rate(vllm_request_ttft_seconds_bucket[5m])
            ) > 10
          for: 3m
          labels:
            severity: critical
          annotations:
            summary: "TTFT p99 exceeds 10s - active incident"
            description: "P99 TTFT at {{ $value | humanizeDuration }}. Users severely impacted."
            runbook_url: "https://runbooks.example.com/llm-d/HighTTFT"
    - name: llm-d.cache
      rules:
        - alert: LowCacheHitRate
          expr: |
            (sum(rate(vllm_cache_hits_total[5m])) /
            (sum(rate(vllm_cache_hits_total[5m])) +
             sum(rate(vllm_cache_misses_total[5m])))) * 100 < 40
          for: 10m
          labels:
            severity: warning
          annotations:
            summary: "KV cache hit rate below 40%"
            description: "Cache hit ratio is {{ $value }}%. Check EPP routing."
    - name: llm-d.resources
      rules:
        - alert: GPUMemoryPressure
          expr: DCGM_FI_DEV_FB_USED / DCGM_FI_DEV_FB_TOTAL * 100 > 90
          for: 5m
          labels:
            severity: warning
          annotations:
            summary: "GPU memory utilization above 90%"
            runbook_url: "https://runbooks.example.com/llm-d/GPUMemoryPressure"
        - alert: HighQueueDepth
          expr: sum(vllm_num_requests_waiting) > 50
          for: 5m
          labels:
            severity: critical
          annotations:
            summary: "Inference queue depth exceeds 50"
        - alert: PodCrashLoop
          expr: |
            increase(kube_pod_container_status_restarts_total{
              namespace="llm-d"
            }[15m]) > 3
          for: 1m
          labels:
            severity: critical
          annotations:
            summary: "Inference pod restarting repeatedly"
            runbook_url: "https://runbooks.example.com/llm-d/PodCrashLoop"
        - alert: HighErrorRate
          expr: |
            sum(rate(vllm_request_errors_total[5m])) /
            sum(rate(vllm_request_total[5m])) * 100 > 1
          for: 5m
          labels:
            severity: warning
          annotations:
            summary: "Request error rate above 1%"

To keep fatigue down, use inhibition rules to suppress warnings when a critical alert is already firing for the same component. If HighTTFTCritical is active, there's zero value in also firing HighTTFTWarning or LowCacheHitRate. Group related alerts so a single notification carries all the context an operator needs. And always put a runbook link in the annotations — one that says not just what the alert means but what to actually do about it. An alert without a runbook is just noise that erodes trust in the monitoring.

09 Alert runbooks

Every alert needs a runbook. Here are step-by-step responses for the three most critical alerts in an llm-d deployment. Follow them in order during an incident — each step narrows the diagnosis until you hit root cause.

HighTTFT (Warning or Critical)

GPUMemoryPressure

PodCrashLoop

10 On-call playbooks

Beyond per-alert runbooks, on-call engineers need playbooks for the messier incidents with multiple correlated symptoms. These five cover the most common llm-d incident patterns. Each one lists the symptoms, diagnostic steps with specific commands and queries, and remediation ranked by urgency.

TTFT Spike Investigation

P99 TTFT exceeds 5 seconds and continues climbing. User complaints about slow initial responses arrive via support channels. The cluster-wide average TTFT has doubled from its baseline within the last 15 minutes.

KV Cache Thrashing

Cache eviction rate spikes to 10x its baseline. Cache hit ratio collapses from 60%+ to below 20% within minutes. TTFT degrades in lockstep with the cache collapse. GPU memory utilization is at or above 95%.

GPU OOM Cascade

One or more pods restart with CUDA OOM errors visible in logs. Remaining pods become overloaded as traffic redistributes, potentially triggering additional OOM kills. Pod restart count increases by 3+ within a 15-minute window.

EPP Routing Imbalance

One pod shows 90%+ GPU utilization while others sit at 30% or below. Queue depth is heavily asymmetric across pods. TTFT varies wildly between requests, with some completing in 200ms and others taking 8+ seconds depending on which pod handles them.

Model Loading Failures

New pods are stuck in Init or CrashLoopBackOff state. Pod logs show model download timeouts, checksum mismatches, or tensor parallel shard connection failures. The cluster has reduced serving capacity because replacement pods cannot start.

11 Distributed tracing for inference

Metrics tell you something is wrong. Traces tell you where and why. Tracing matters for llm-d because a single inference request crosses multiple components, each with its own latency contribution and failure modes. To understand the full request lifecycle you need end-to-end traces that stitch the Gateway, EPP, and vLLM pods into one coherent story.

A request through an llm-d cluster has seven distinct stages, and each should be a span in your trace. First span: the Gateway — TLS termination, auth, rate limiting, request validation. Usually 1-5ms, but misconfigured rate limiters or cert validation issues can blow it up dramatically. Second: EPP scoring, where the Endpoint Picker Policy evaluates every candidate pod on cache state, load, queue depth, and health. Usually sub-millisecond, but it grows when the candidate pool is large or scoring does heavy cache lookups. Third: pod selection and request forwarding, including any network latency between the EPP and the chosen pod.

The fourth span is where most of the latency lives: prefill computation. Annotate it with input token count, tokens served from the KV cache, and tokens computed fresh. That annotation is the difference between "slow because it had a 50K-token context" and "slow because it missed the cache and recomputed 50K tokens that should have been cached." Fifth: KV cache transfer, which matters when llm-d runs disaggregated prefill/decode and ships cache state between specialized prefill and decode pods over NVLink or RDMA. Sixth: the decode phase, tokens generated autoregressively one at a time. Seventh: response streaming back through the Gateway to the client.

OpenTelemetry is the standard here. vLLM supports it natively, and the llm-d control plane components can be instrumented with the OTel SDKs for Go and Python. The one hard requirement: propagate trace context across component boundaries with W3C Trace Context headers, so a single trace spans the whole lifecycle from Gateway ingress to client response.

# OpenTelemetry Collector config for llm-d tracing
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 5s
    send_batch_size: 1024
  attributes:
    actions:
      - key: deployment.environment
        value: production
        action: upsert

exporters:
  otlp/jaeger:
    endpoint: jaeger-collector.observability:4317
    tls:
      insecure: true

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch, attributes]
      exporters: [otlp/jaeger]

Tracing really pays off when you correlate traces with metrics. A TTFT spike on the Grafana dashboard becomes actionable the moment you can click through to the exact traces behind the p99 and see they all missed the cache on one pod, or all spent 3 seconds in the EPP queue because a health check was returning stale data. That link between aggregate metrics and individual traces is what turns "something is wrong somewhere" into "here's exactly what broke, on which component, and why."

12 OpenTelemetry setup for vLLM

Here's a concrete end-to-end example: instrumenting a vLLM deployment with OpenTelemetry and exporting traces to Jaeger. Three pieces: enable OTel in vLLM itself, deploy the OpenTelemetry Collector as a sidecar or DaemonSet, and deploy Jaeger for visualization.

Enabling OTel in vLLM

vLLM supports OpenTelemetry via environment variables — set these on your deployment and it emits traces with zero code changes.

env:
  # Enable OpenTelemetry tracing in vLLM
  - name: OTEL_EXPORTER_OTLP_ENDPOINT
    value: "http://otel-collector.observability:4317"
  - name: OTEL_SERVICE_NAME
    value: "vllm-inference"
  - name: OTEL_TRACES_EXPORTER
    value: "otlp"
  - name: OTEL_EXPORTER_OTLP_PROTOCOL
    value: "grpc"
  # Propagate trace context across services
  - name: OTEL_PROPAGATORS
    value: "tracecontext,baggage"
  # Sample 10% of traces in production to control volume
  - name: OTEL_TRACES_SAMPLER
    value: "parentbased_traceidratio"
  - name: OTEL_TRACES_SAMPLER_ARG
    value: "0.1"

Deploying Jaeger for trace visualization

For production, deploy Jaeger with persistent storage. The minimal all-in-one deployment below is fine for staging; in production, use the Jaeger Operator with Elasticsearch or Cassandra behind it.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: jaeger
  namespace: observability
spec:
  replicas: 1
  selector:
    matchLabels:
      app: jaeger
  template:
    metadata:
      labels:
        app: jaeger
    spec:
      containers:
        - name: jaeger
          image: jaegertracing/all-in-one:1.57
          ports:
            - containerPort: 16686  # Jaeger UI
            - containerPort: 4317   # OTLP gRPC
            - containerPort: 4318   # OTLP HTTP
          env:
            - name: COLLECTOR_OTLP_ENABLED
              value: "true"
---
apiVersion: v1
kind: Service
metadata:
  name: jaeger-collector
  namespace: observability
spec:
  selector:
    app: jaeger
  ports:
    - name: otlp-grpc
      port: 4317
      targetPort: 4317
    - name: otlp-http
      port: 4318
      targetPort: 4318
    - name: ui
      port: 16686
      targetPort: 16686

With this in place, vLLM sends traces via OTLP gRPC to the OpenTelemetry Collector, which batches, enriches, and forwards them to Jaeger. Open the Jaeger UI on port 16686 and you'll see a trace per inference request, with spans for prefill, decode, and response streaming. From Grafana you can link straight to Jaeger traces using the traceID exemplar label vLLM attaches to its Prometheus histograms — metric anomaly to individual trace in one click.

13 GPU topology monitoring

Modern GPU servers aren't homogeneous compute pools. The physical interconnect topology between GPUs dramatically affects inference performance — it's why llm-d's scheduler is topology-aware in the first place. Monitoring topology is how you diagnose the performance anomalies that have no obvious cause at the application layer and that standard Kubernetes metrics can't even represent.

The big distinction is NVLink vs. PCIe. NVLink gives you over 600 GB/s of bidirectional bandwidth between directly connected GPUs in the same NVLink domain; PCIe Gen5 x16 gives you roughly 32 GB/s. Nearly a 20x difference. When tensor parallelism splits a large model across GPUs, the all-reduce operations that sync intermediate activations between them are extremely bandwidth-hungry. Over NVLink, that communication is negligible next to the compute. Over PCIe — because the scheduler put tensor-parallel shards in different NVLink domains — the communication can dominate the compute entirely, especially at smaller batch sizes where the compute-to-communication ratio is already unfavorable.

NUMA (Non-Uniform Memory Access) topology adds another layer. On multi-socket servers, GPUs hang off specific CPU sockets via PCIe root complexes, and touching GPU memory or doing DMA from the wrong NUMA node costs extra latency through inter-socket traffic over UPI or Infinity Fabric. llm-d's scheduler should keep tensor-parallel groups on GPUs sharing the same NVLink domain and NUMA node. Monitoring is how you confirm placement is right — and how you catch violations, like a pod restart landing on a node where the free GPU isn't NVLink-connected to its tensor-parallel peers.

The DCGM (Data Center GPU Manager) exporter is the standard tool for getting GPU-level metrics into Prometheus. Beyond basic utilization and memory, it exposes NVLink-specific counters that are gold for topology monitoring and hardware health.

# Key DCGM metrics for topology monitoring
DCGM_FI_DEV_NVLINK_BANDWIDTH_TOTAL       # NVLink aggregate throughput
DCGM_FI_DEV_NVLINK_CRC_FLIT_ERROR_COUNT  # NVLink CRC flit errors
DCGM_FI_DEV_NVLINK_CRC_DATA_ERROR_COUNT  # NVLink CRC data errors
DCGM_FI_DEV_NVLINK_REPLAY_ERROR_COUNT    # NVLink replay errors
DCGM_FI_DEV_PCIE_TX_THROUGHPUT           # PCIe transmit throughput
DCGM_FI_DEV_PCIE_RX_THROUGHPUT           # PCIe receive throughput

Take NVLink errors seriously, immediately. A rising count of CRC errors or replay errors on an NVLink connection can mean a failing GPU, a loose physical connection in the NVLink bridge, or thermal issues degrading signal integrity. The retransmissions cut effective bandwidth and add latency jitter that's nearly impossible to diagnose from application-level metrics. Left unmonitored, a degrading link shows up as intermittent latency spikes that look random at the application layer but have a very non-random hardware cause. Alert on any non-zero rate of NVLink errors and go look right away. Hardware doesn't fix itself — it gets worse until the link dies entirely.

llm-d uses topology actively when scheduling. New model replicas go to GPU groups fully connected via NVLink within one domain. KV cache transfers for disaggregated prefill/decode prefer paths over NVLink rather than PCIe to keep transfer latency down. Monitor those decisions and correlate them with observed inference latency — that's how you verify topology-aware scheduling is actually paying off instead of just adding complexity.

14 Cost observability

GPUs are the most expensive thing in the stack. An H100 SXM instance runs roughly $30-40 per hour on the major clouds, and at eight or sixteen of these machines, every single percentage point of utilization efficiency is thousands of dollars a month — saved or wasted. Cost observability isn't a nice-to-have. It's the layer that tells you whether your LLM serving is financially sustainable at scale or whether you're quietly hemorrhaging money on underutilized hardware.

The fundamental metric is cost-per-token. The math is simple in principle: total GPU cost per hour divided by tokens generated per hour gives you the marginal cost of each token. Track it continuously and break it down by model, LoRA adapter, and request type so you can see where the GPU dollars actually go and where they're being wasted.

# Cost-per-million-tokens calculation
# Assume $35/hr per H100 GPU, 8 GPUs in cluster
# GPU cost per second
gpu_cost_per_second = 8 * 35 / 3600  # $0.0778/second

# Tokens generated per second (from Prometheus)
# sum(rate(vllm_tokens_generated_total[5m]))
# Example: 2,400 tokens/second

# Cost per million tokens
# ($0.0778 / 2400) * 1,000,000 = $32.41 per million tokens

Cost observability is also where cache-aware routing — the core economic argument for llm-d existing at all — becomes quantifiable. A cache hit skips the prefill phase entirely, and prefill is the most compute-intensive part of inference, often 10-50x more GPU cycles per token than decode. Routing requests to pods that already cached the prefix eliminates redundant prefill, which cuts GPU-seconds per request, which cuts cost-per-token.

Concrete example: a RAG app that prepends a 10,000-token document context to every user query. Without cache-aware routing, every request pays full prefill on those 10,000 tokens no matter which pod handles it. With llm-d routing to pods that already cached the document context, only the user query portion — maybe 50-200 tokens — needs fresh prefill. That's a 98% reduction in prefill compute for the repeated prefix. At scale, across thousands of requests per hour, that means running the same workload on 30-40% fewer GPUs, or serving 50-70% more traffic on the same hardware, with no hit to latency or quality.

Track three things on the cost dashboard. First, raw cost per million tokens over time, split by model and adapter — your unit economics, and the number your finance team cares about. Second, estimated savings from cache hits, calculated by comparing prefill tokens actually computed against the total that would have needed prefill without caching — the dollar value of your routing intelligence. Third, GPU utilization efficiency: the ratio of time GPUs spend on useful inference versus idle, overhead, or redundant computation that caching should have eliminated. A well-tuned llm-d cluster should hit 70-80%. Below 50% means you're paying for GPU time that isn't producing tokens — go figure out whether it's scheduling inefficiency, memory management overhead, or just not enough traffic to keep the hardware busy.

15 Grafana dashboard JSON

You don't have to build panels from scratch — you can import dashboard JSON directly into Grafana. This snippet defines a TTFT panel you can paste into any dashboard's JSON model: the three percentile queries (p50, p95, p99), color thresholds, and the time series config.

To use it: open your Grafana dashboard, click the gear icon, pick "JSON Model," and add this panel definition to the panels array. Adjust gridPos to position it where you want.

{
  "id": 1,
  "type": "timeseries",
  "title": "Time-to-First-Token (TTFT)",
  "description": "P50, P95, P99 TTFT across all inference pods",
  "gridPos": { "h": 8, "w": 12, "x": 0, "y": 0 },
  "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" },
  "targets": [
    {
      "refId": "A",
      "expr": "histogram_quantile(0.50, rate(vllm_request_ttft_seconds_bucket[5m]))",
      "legendFormat": "p50"
    },
    {
      "refId": "B",
      "expr": "histogram_quantile(0.95, rate(vllm_request_ttft_seconds_bucket[5m]))",
      "legendFormat": "p95"
    },
    {
      "refId": "C",
      "expr": "histogram_quantile(0.99, rate(vllm_request_ttft_seconds_bucket[5m]))",
      "legendFormat": "p99"
    }
  ],
  "fieldConfig": {
    "defaults": {
      "unit": "s",
      "thresholds": {
        "mode": "absolute",
        "steps": [
          { "color": "green", "value": null },
          { "color": "yellow", "value": 3 },
          { "color": "red", "value": 5 }
        ]
      },
      "custom": {
        "lineWidth": 2,
        "fillOpacity": 10,
        "gradientMode": "scheme",
        "showPoints": "never",
        "spanNulls": true
      }
    },
    "overrides": [
      {
        "matcher": { "id": "byName", "options": "p50" },
        "properties": [
          { "id": "color", "value": { "fixedColor": "#73BF69", "mode": "fixed" } }
        ]
      },
      {
        "matcher": { "id": "byName", "options": "p95" },
        "properties": [
          { "id": "color", "value": { "fixedColor": "#FADE2A", "mode": "fixed" } }
        ]
      },
      {
        "matcher": { "id": "byName", "options": "p99" },
        "properties": [
          { "id": "color", "value": { "fixedColor": "#F2495C", "mode": "fixed" } }
        ]
      }
    ]
  },
  "options": {
    "tooltip": { "mode": "multi", "sort": "desc" },
    "legend": { "displayMode": "table", "placement": "bottom", "calcs": ["mean", "max"] }
  }
}

16 Dashboard design patterns

Building dashboards is easy. Building dashboards someone can actually parse in seconds during a 2am incident, tired and under pressure, is a completely different problem. The difference is design patterns that respect how humans process visual information under stress. When the cluster is degrading and users are complaining, the on-call needs to understand the situation in seconds — not minutes of scrolling through fifty undifferentiated panels.

The four golden signals for LLM inference

Google's Four Golden Signals — latency, traffic, errors, saturation — map naturally onto LLM inference, but the specific metrics behind each signal are completely different from a traditional web service and worth choosing carefully.

• Latency is TTFT plus ITL. TTFT captures the initial wait that determines whether users perceive the system as responsive. ITL captures the streaming experience that determines whether the generation feels smooth or stuttery. Display both prominently on the overview dashboard, with TTFT as the primary signal and ITL as supporting context that aids in diagnosis.
• Traffic is requests per second plus tokens per second. Requests per second tells you how many users are being served. Tokens per second tells you how much actual computational work the cluster is performing. These can diverge significantly in practice: a few long-context requests generate enormous token volumes while the request count stays low, and conversely, many short requests can produce high request rates with modest token throughput.
• Errors are HTTP 5xx responses plus OOM kills. HTTP 5xx errors indicate application-level failures such as timeouts, model loading failures, or input validation errors. OOM kills indicate infrastructure-level failures where the GPU or system memory is exhausted. Track them separately because they have completely different root causes and different remediation strategies.
• Saturation is GPU memory utilization plus queue depth. GPU memory saturation triggers KV cache evictions that cascade into latency degradation. Queue saturation indicates insufficient compute capacity for the current traffic load. Both are leading indicators of user-facing latency problems: they spike before TTFT degrades, giving you a window to intervene proactively before users notice.

Dashboard hierarchy and the RED method

Organize dashboards in a strict drill-down hierarchy that mirrors how you actually investigate. Top level: the Four Golden Signals for the whole cluster, four to six big panels readable from across the room. Something looks wrong? Click the anomalous signal and land on a second-level dashboard showing that signal decomposed by pod, model, or LoRA adapter. A specific pod or model misbehaving? Click again for a third-level dashboard with that entity's detailed metrics — DCGM counters, per-request traces, historical patterns.

The RED method (Rate, Errors, Duration) works really well for per-model views in multi-model deployments. For each model or adapter, show request rate, error rate, and TTFT distribution side by side. You instantly see which models are struggling and which are healthy, and it doubles as a capacity planning tool: if one model's request rate is growing 20% month-over-month while the others are flat, you know exactly where the next GPUs go before growth causes saturation.

Panels, time ranges, and visual hierarchy

Use the F-pattern for layout: the most critical info goes top-left, because eye-tracking research consistently shows that's where people look first. For llm-d, that means TTFT p99 top-left, error rate to its right, traffic and saturation on the second row. Detailed breakdowns and supporting metrics belong below the fold — available for investigation, not competing for attention during triage.

For time ranges, default the overview dashboard to 6 hours — enough history to see trends and spot when a degradation started, without drowning the display in noise. During active incidents, switch to 1-hour or 30-minute views for fine-grained changes. For weekly capacity reviews, use 7-day or 30-day views for growth trends and recurring patterns. And put deployment annotations as vertical lines on every time-series chart so metric changes correlate instantly with code, config, or model version deploys. New model version, EPP scoring weight update, eviction policy change, cluster scale event — every one of those should be a visible marker on every dashboard's time axis.

Keep color coding consistent across every dashboard in the hierarchy. Green healthy, amber warning, red critical — using the exact same thresholds as your alert rules, so dashboard colors match alert states precisely. When a panel goes red on the overview, the operator should know for certain that a corresponding alert has fired or is about to within its for duration. That consistency between visuals and alert state builds trust in the monitoring, and trust is what decides whether operators actually use dashboards during incidents or abandon them for ad-hoc log grepping and gut instinct.

17 Dashboard anti-patterns (and a layout that works)

Even well-intentioned dashboards can undermine observability if they fall into the common traps. These anti-patterns show up over and over in LLM inference monitoring setups, and each one hurts most at exactly the wrong moment: mid-incident, under time pressure.

Anti-pattern: the wall of charts

Cramming 40+ panels onto a single dashboard is cognitive overload. Mid-incident, operators scroll endlessly hunting for the relevant panel, and the sheer density makes it impossible to separate signal from noise. The fix: cap overview dashboards at 8-12 panels. Move detailed breakdowns to drill-down dashboards linked from the overview. Every panel should be big enough to read without squinting, with a title that states the question it answers.

Anti-pattern: wrong default time ranges

Defaulting everything to a 24-hour range buries acute incidents in historical data — a TTFT spike that started 10 minutes ago is barely visible on a 24-hour chart. Defaulting to 5 minutes has the opposite problem: no context for when the degradation started. The fix: 1 hour default for operational dashboards, 6 hours for overviews. And train operators to adjust the range as part of incident response — zoom in to isolate the start, zoom out to spot patterns.

Anti-pattern: missing annotations

Without deployment markers, scaling events, and incident annotations on time-series charts, operators can't connect metric changes to operational events. A sudden TTFT increase might line up perfectly with a model version deploy, but without an annotation marking it, someone has to dig through deployment logs manually to discover the correlation. The fix: annotate every deployment, config change, scaling event, and incident start/end. Grafana supports annotations via its HTTP API, and most CI/CD pipelines can push one as a post-deploy step.

Anti-pattern: inconsistent color thresholds

If one panel uses green below 50% and another uses green below 80% for the same metric type, operators lose the ability to scan dashboards by color. Color should communicate severity instantly — no reading axis labels and doing mental math about whether a value is concerning. The fix: define a color standard (for example, green below 60% GPU utilization, amber 60-85%, red above 85%) and apply it consistently everywhere that metric appears.

Anti-pattern: mixing operational and business metrics

Cost-per-token panels next to GPU utilization panels on the same dashboard serve two different audiences with conflicting time horizons. Ops needs real-time health signals; finance needs daily and weekly cost trends. Mixed together, the dashboard satisfies neither. The fix: separate operational dashboards (latency, errors, saturation) from business dashboards (cost, utilization efficiency, capacity planning). Link between them for context, but don't merge them into one view.

A layout that works: the llm-d operations console

The layout I'd recommend for an llm-d operations dashboard is four rows, each answering one category of question. It's been refined through real incident response and balances overview against detail.

Row 1: Health Overview (4 stat panels, single row)
  [TTFT p99: stat]  [Error Rate %: stat]  [Cache Hit %: stat]  [Active Pods: stat]

Row 2: Latency Detail (2 time-series panels)
  [TTFT p50/p95/p99 over time]          [ITL p50/p95/p99 over time]

Row 3: Resource Utilization (3 panels)
  [GPU Memory % by pod: heatmap]  [Queue Depth by pod: bar]  [Tokens/sec: timeseries]

Row 4: Routing and Cache (2 panels)
  [EPP routing distribution: stacked bar]  [Cache eviction rate: timeseries]

The whole thing fits on a single screen at 1920x1080 without scrolling. Row 1 gives instant health assessment via color-coded stat panels. Row 2 gives latency context for diagnosis. Row 3 shows resource state. Row 4 exposes routing and cache behavior. During an incident you read top to bottom: confirm what's broken (row 1), understand the latency impact (row 2), check resource constraints (row 3), then dig into routing as a root cause (row 4).

18 Dashboard starter kit

Starting Grafana dashboards from a blank page is rough. These three layouts cover the most common operational needs for an llm-d cluster. Each answers a specific category of question and can be built from the PromQL in earlier sections. Treat them as blueprints: start with the recommended widgets, then adapt to your team's workflow.

19 Bringing it all together

LLM inference observability isn't something you solve by installing a Helm chart and walking away. It takes understanding the computational quirks of autoregressive token generation, the cascading failure modes of KV cache management under memory pressure, the cost dynamics of GPU compute at scale, and the topology constraints of modern multi-GPU servers. Standard Kubernetes monitoring won't save you. It'll actively mislead you — reporting healthy dashboards while your users sit through multi-second latencies caused by problems your monitoring stack can't even represent.

The stack in this post — Prometheus for metrics, Grafana for dashboards, OpenTelemetry for distributed tracing, DCGM for GPU topology and hardware health, and PromQL alert rules calibrated for LLM workloads — gives you real visibility into what your llm-d cluster is doing and why. It surfaces the metrics that actually determine user experience (TTFT percentiles, KV cache hit ratio, cost-per-token) instead of the ones that are easy to collect but misleading to read (aggregate CPU utilization, pod count).

Start with the metrics foundation: instrument your vLLM pods with the ServiceMonitor config above. Build the four essential dashboards — cluster overview, pod detail, cache performance, cost analysis. Set up the alert rules with thresholds tuned to your hardware and SLAs. Then add distributed tracing to see per-request lifecycles and routing decisions. Then add GPU topology monitoring to catch hardware degradation before it cascades into application failures. Each layer builds on the previous one, and each catches a class of problems the layers below can't. The gap between collecting data and understanding your system is the gap between monitoring and observability — and for LLM inference at scale, honestly, understanding isn't optional.