llm-d on OpenShift AI
Enterprise LLM Inference on OpenShift AI
llm-d brings production-grade, GPU-optimized large language model serving to Red Hat OpenShift AI with intelligent routing, disaggregated serving, and enterprise compliance built in.
7-Layer Inference Stack
From user API calls to GPU silicon, every layer is purpose-built for high-throughput, low-latency LLM serving on OpenShift.
- User / API Gateway -- ingress and auth
- OpenShift Route -- TLS termination and load balancing
- Red Hat AI Operator v3.4 -- lifecycle management
- KServe + LLMInferenceService CRD
- Endpoint Picker (EPP v0.5.x) -- intelligent routing
- vLLM v0.8.4 pods -- model serving runtime
- GPU Hardware -- H100, H200, B200, MI300X
KV Cache-Aware Endpoint Picker
EPP v0.5.x scores every vLLM replica in real time, routing each request to the pod with the warmest KV cache, lowest queue depth, and matching LoRA adapter.
Score = 3 * prefix-cache
+ 2 * queue-depth
+ 2 * active-requests
- KV-cache affinity -- reuse computed attention
- LoRA locality -- route to adapter-loaded pods
- Prefix cache routing -- shared prompt optimization
- Queue-depth awareness -- avoid hot pods
Prefill / Decode Separation
NIXL enables zero-copy KV cache transfers between dedicated prefill and decode GPU pools. Each pool scales independently to match workload demands.
- Prefill pool -- compute-optimized for prompt processing
- Decode pool -- memory-optimized for token generation
- NIXL interconnect -- zero-copy KV cache transfer
- Independent autoscaling per phase
Certified GPU Ecosystem
Five GPU families certified, three major platform releases, and the latest vLLM runtime.
GPU Hardware Matrix
- H100 SXM -- 80GB, 3.35 TB/s
- H100 PCIe -- 80GB, 2.0 TB/s
- H200 SXM -- 141GB, 4.8 TB/s
- B200 -- 192GB, 8.0 TB/s
- MI300X -- 192GB, 5.3 TB/s
Red Hat AI v3.4
Latest release -- June 2026
- LLMInferenceService CRD
- Disaggregated prefill / decode
- FIPS 140-3 validated
- vLLM v0.8.4 runtime
- B200 GPU support
Version Evolution
- v3.0 -- Oct 2025: vLLM v0.6.x, initial GA
- v3.3 -- Feb 2026: EPP v0.3.x, AMD MI300X TP
- v3.4 -- Jun 2026: LLMInferenceService, P/D split, FIPS
Production-Grade Features
Security, multi-tenancy, and autoscaling built for the most demanding enterprise workloads.
Security & Compliance
Validated for the most demanding regulatory environments.
- FIPS 140-3 cryptographic validation
- SOC2 Type II attestation
- HIPAA and PCI-DSS compliant
- FedRAMP authorization
- DISA STIG hardened
Multi-Tenancy & LoRA
Shared base models with per-team LoRA adapters for cost-efficient customization.
- Shared base model across tenants
- Per-team LoRA adapter routing
- EPP routes by adapter ID
- 40% cost savings vs. dedicated models
Scale-to-Zero Autoscaling
Weighted Virtual Autoscaler manages independent prefill and decode scaling.
- Scale to zero on idle -- no wasted GPU hours
- Independent prefill / decode scaling
- WVA-driven replica management
- 35% infrastructure cost savings
From kubectl to Live Traffic
A single kubectl apply triggers a 9-step automated flow. The AI Operator reconciles your LLMInferenceService CRD into a fully routed, health-checked inference endpoint.
- kubectl apply -- submit CRD (<1s)
- Controller reconcile -- validate and plan
- InferencePool creation -- resource allocation
- Pod scheduling -- GPU node placement
- Model loading -- weights to GPU memory (30s -- 5min)
- Health checks -- readiness and liveness probes
- EPP registration -- pod joins scoring pool
- Route created -- external endpoint live
- Traffic flowing -- requests served
Why OpenShift AI + llm-d
The only platform that combines KV-cache-aware routing, disaggregated serving, multi-tenant LoRA, and FIPS compliance in a single CRD-driven stack.
| Capability | OpenShift AI + llm-d | Raw Kubernetes | Cloud Managed |
|---|---|---|---|
| KV-cache-aware routing | Yes -- EPP v0.5.x | No | Varies |
| Disaggregated P/D | Yes -- NIXL | Manual | No |
| LoRA multi-tenant routing | Yes -- built-in | Custom code | Limited |
| FIPS 140-3 / FedRAMP | Yes -- certified | DIY | Provider-dependent |
| Scale-to-zero autoscaling | Yes -- WVA | HPA only | Yes |
| GPU families certified | 5 families | Self-managed | 2--3 families |
| Operational overhead | CRD-driven | High -- 2.5 FTE | Managed |