llm-d Monitoring Dashboard | Red Hat OpenShift

▼

Key Metrics

TTFT p95 vllm:time_to_first_token_seconds

82ms

Time to first token (95th percentile)

Prefix Cache Hit Rate vllm:prefix_cache_hits

94.2%

Token-level cache hit ratio

Token Throughput vllm:generation_tokens

1,247 tok/s

Generation tokens per second

Pool Saturation llm_d_epp_flow_control_pool_saturation

0.34

0.0 = empty, 1.0 = full, >1.0 = throttling

▼

Inference Performance

TTFT Over Time (per pod) histogram_quantile(0.95, ...)

Pod 1

Pod 2

Pod 3

Prefix Cache Hit Rate rate(vllm:prefix_cache_hits[5m]) / rate(...queries[5m])

Pod 1

Pod 2

Pod 3

▼

EPP Routing & Scheduling

Prefix Indexer Hit Ratio llm_d_epp_prefix_indexer_hit_ratio

Hit Ratio (avg)

Per-Pod Queue Depth llm_d_epp_per_endpoint_queue_size

Pod 1

Pod 2

Pod 3

▼

KV Cache Utilization

KV Cache Usage Per Pod vllm:kv_cache_usage_perc

42%

Pod 1

38%

Pod 2

35%

Pod 3

Requests Running vs Waiting vllm:num_requests_running / waiting

Running

Waiting

▼

GPU Infrastructure (NVIDIA DCGM)

GPU Utilization DCGM_FI_DEV_GPU_UTIL

67%

H200 #0

62%

H200 #1

59%

H200 #2

GPU Memory (FB Used) DCGM_FI_DEV_FB_USED

72%

H200 #0

71%

H200 #1

70%

H200 #2

GPU Temperature DCGM_FI_DEV_GPU_TEMP

H200 #0

H200 #1

H200 #2

Enable User Workload Monitoring

Patch the cluster monitoring ConfigMap to enable Prometheus for user-defined projects.

apiVersion: v1
kind: ConfigMap
metadata:
  name: cluster-monitoring-config
  namespace: openshift-monitoring
data:
  config.yaml: |
    enableUserWorkload: true

ServiceMonitor for EPP

Scrape the EPP metrics endpoint (port 9090). The EPP exposes ~50 metrics with the llm_d_epp_* prefix.

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: llm-d-epp
  namespace: llm-serving
spec:
  selector:
    matchLabels:
      app: llm-d-epp
  endpoints:
    - port: metrics
      path: /metrics
      interval: 15s

PodMonitor for vLLM

Scrape vLLM's /metrics endpoint (port 8000). vLLM exposes ~386 metrics including TTFT, cache hit rates, and KV cache utilization.

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: vllm-pods
  namespace: llm-serving
spec:
  selector:
    matchLabels:
      app: vllm-server
  podMetricsEndpoints:
    - port: http
      path: /metrics
      interval: 15s

DCGM via GPU Operator

Enable the DCGM ServiceMonitor in the GPU Operator ClusterPolicy, then label the namespace for platform Prometheus discovery.

apiVersion: nvidia.com/v1
kind: ClusterPolicy
metadata:
  name: gpu-cluster-policy
spec:
  dcgmExporter:
    enabled: true
    serviceMonitor:
      enabled: true

# Then label the namespace:
# oc label ns/nvidia-gpu-operator \
#   openshift.io/cluster-monitoring=true

Essential PromQL Queries for llm-d Monitoring

Metric	PromQL	What it tells you
TTFT p95	histogram_quantile(0.95, sum by (le) (rate(vllm:time_to_first_token_seconds_bucket[5m])))	95th percentile time-to-first-token. Healthy: <200ms with cache hits. Spikes mean cache misses or queue pressure.
Cache Hit Rate	rate(vllm:prefix_cache_hits[5m]) / clamp_min(rate(vllm:prefix_cache_queries[5m]), 1)	Token-level cache hit ratio. Healthy: >80% for stable workloads. Below 50% means routing isn't concentrating prefixes.
KV Cache Usage	vllm:kv_cache_usage_perc	Fraction of KV cache blocks in use. Above 90% triggers LRU eviction. Above 95% degrades hit rate.
Token Throughput	sum(rate(vllm:generation_tokens[5m]))	Tokens generated per second across all pods. Drop means pods are stuck in prefill or queue is backing up.
Queue Depth	vllm:num_requests_waiting	Requests waiting for scheduler capacity. Sustained >10 means you need more replicas or GPU memory.
EPP Hit Ratio	histogram_quantile(0.5, rate(llm_d_epp_prefix_indexer_hit_ratio_bucket[5m]))	Median prefix match quality. 1.0 = full prefix match. Below 0.5 means the EPP is routing to cold pods.
Pool Saturation	llm_d_epp_flow_control_pool_saturation	EPP flow control. Above 1.0 means requests are being throttled. 0.3–0.7 is healthy.
GPU Utilization	DCGM_FI_DEV_GPU_UTIL	GPU SM utilization %. Can be misleading for decode (memory-bound). Pair with TTFT and throughput.
GPU Memory	DCGM_FI_DEV_FB_USED / (DCGM_FI_DEV_FB_USED + DCGM_FI_DEV_FB_FREE)	GPU framebuffer memory usage. Model + KV cache. Above 95% risks OOM.
SLO Violations	rate(llm_d_epp_request_slo_violation_total[5m])	TTFT or TPOT SLO breaches per second. Any sustained rate means your SLO thresholds are being exceeded.

PrometheusRule -- llm-d Alert Rules

Drop this into your llm-serving namespace. Alerts appear automatically in the OpenShift Console under Observe → Alerting.

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: llm-d-alerts
  namespace: llm-serving
spec:
  groups:
  - name: llm-d.inference
    rules:

    # TTFT SLO breach
    - alert: HighTTFTp95
      expr: |
        histogram_quantile(0.95,
          sum by (le) (rate(
            vllm:time_to_first_token_seconds_bucket[5m]
          ))
        ) > 2.0
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "TTFT p95 above 2s for 5 minutes"

    # Cache hit rate degradation
    - alert: LowPrefixCacheHitRate
      expr: |
        rate(vllm:prefix_cache_hits[10m])
        / clamp_min(rate(vllm:prefix_cache_queries[10m]), 1)
        < 0.5
      for: 10m
      labels:
        severity: warning
      annotations:
        summary: "Cache hit rate below 50%"

    # KV cache near full
    - alert: HighKVCacheUtilization
      expr: vllm:kv_cache_usage_perc > 0.92
      for: 5m
      labels:
        severity: critical
      annotations:
        summary: "KV cache above 92% on {{ $labels.pod }}"

    # EPP pool saturated (throttling)
    - alert: EPPPoolSaturated
      expr: llm_d_epp_flow_control_pool_saturation > 1.0
      for: 3m
      labels:
        severity: critical
      annotations:
        summary: "EPP is throttling requests"

    # GPU temperature critical
    - alert: GPUTemperatureCritical
      expr: DCGM_FI_DEV_GPU_TEMP > 85
      for: 5m
      labels:
        severity: critical
      annotations:
        summary: "GPU temp {{ $value }}C on {{ $labels.gpu }}"

    # High error rate
    - alert: HighInferenceErrorRate
      expr: |
        rate(llm_d_epp_request_error_total[5m])
        / clamp_min(rate(llm_d_epp_request_total[5m]), 1)
        > 0.05
      for: 5m
      labels:
        severity: critical
      annotations:
        summary: "Inference error rate above 5%"