Inside the llm-d production inference cluster. Real-time metrics, GPU topology, intelligent routing, and full-stack tracing for LLM serving at scale.
A DGX H100 node running 4 inference pods serving Llama 3.1 70B with live LoRA adapter hot-swapping, prefix-aware caching, and sub-second time-to-first-token.
Two NUMA domains, each with 4 H100 80GB GPUs connected via NVLink 4.0, bridged by NVSwitch at 450 GB/s bidirectional throughput.
Continuously streamed metrics across all inference pods. From time-to-first-token latency to prefix cache hit ratios.
Every request entering the Inference Gateway (IGW) is scored by the Endpoint Picker (EPP) using KV-cache-aware, load-aware scheduling. Prefix-cache affinity, current load, queue depth, and pod health all factor in. Weights below are illustrative.
Every inference request is traced through six stages. From gateway ingress through EPP routing, prefill, KV transfer (when prefill and decode are disaggregated), token decode, and response delivery.
The cluster detects anomalies in real time. Latency spikes, GPU OOM events, pod crashes, and cache storms, with structured remediation guidance for every alert.
llm-d brings transparent, production-grade observability to LLM inference. From GPU silicon to application response. Every layer visible. Every decision traceable. Every anomaly actionable.