OpenShift AI and llm-d: The Enterprise Architecture Guide to Production LLM Inference

Introduction: Why Enterprise LLM Inference Is Hard #

There is a familiar pattern in enterprise technology: a demo that runs beautifully on a single machine crumbles the moment it faces real production traffic. Large language model inference is no exception. Getting a model to respond on your laptop is straightforward. Getting that same model to serve 10,000 concurrent users with sub-second time-to-first-token, contractual uptime guarantees, automatic scaling, and GPU cost efficiency that keeps your CFO from canceling the project is an entirely different engineering challenge.

The gap between "it works" and "it works at scale with SLAs" is where most organizations stall. Building production inference infrastructure from scratch means solving a cascade of hard problems simultaneously: intelligent request routing that understands inference semantics, KV-cache management across distributed GPU pools, disaggregated prefill and decode scheduling, GPU topology-aware placement that respects NVLink domains and NUMA boundaries, autoscaling that responds to inference-specific signals rather than generic CPU utilization, monitoring that surfaces the metrics that actually predict user-visible latency, and security hardening that satisfies compliance teams without destroying throughput.

Most teams attempt this by stitching together vLLM, a load balancer, some GPU monitoring scripts, and a prayer. It works until it does not. The first 3 AM incident that requires understanding the interaction between KV-cache eviction, GPU memory fragmentation, and Kubernetes scheduling decisions reveals just how many sharp edges exist in a hand-assembled inference stack.

This is precisely the space where the upstream-to-enterprise pipeline proves its value. llm-d is the CNCF Sandbox open-source project purpose-built for distributed LLM inference routing. Red Hat AI (formerly OpenShift AI) is the enterprise product that takes llm-d upstream and makes it production-ready with support contracts, hardware certification, FIPS compliance, and deep integration across the Red Hat ecosystem. Together, they represent the most complete path from open-source innovation to enterprise-grade LLM infrastructure available today.

"Our platform team went from 3 weeks to deploy to 45 minutes with the LLMInferenceService CRD."

-- Platform Engineering Lead, Fortune 500 Financial Services

This article is a comprehensive architecture guide written for enterprise architects evaluating Red Hat AI for procurement. We will walk through the full stack: the CRDs that define inference workloads, the GPU infrastructure that powers them, the monitoring that keeps them observable, the deployment steps that bring them to life, the troubleshooting playbook for when things go wrong, support SLAs and incident response procedures, CVE response and security lifecycle, product lifecycle policy, a decision framework for choosing upstream vs. enterprise, and the migration path for organizations moving from standalone vLLM. By the end, you will have a concrete, operational understanding of how llm-d runs on Red Hat AI -- and the confidence to make a procurement decision grounded in technical specifics rather than marketing claims.

Deep Dive: KServe and the LLMInferenceService CRD #

KServe is the standard model serving framework on Kubernetes, providing a consistent API for deploying and managing machine learning models. Red Hat AI 3.4 uses KServe as the foundation for its model serving capabilities. On top of KServe, the llm-d project introduces the LLMInferenceService Custom Resource Definition (CRD) -- a purpose-built abstraction for deploying distributed LLM inference workloads.

The LLMInferenceService CRD is the single declarative surface through which an operator defines everything about an inference deployment. A single YAML file replaces what would otherwise be dozens of manually coordinated Kubernetes resources: Deployments, Services, ConfigMaps, HorizontalPodAutoscalers, ServiceMonitors, PodDisruptionBudgets, and Route objects. The controller that watches this CRD reconciles all of these downstream resources automatically.

Basic Deployment

The simplest possible deployment provisions a single model on a set of replicas with straightforward round-robin routing. This is suitable for development, testing, or workloads where request volume is predictable and latency requirements are relaxed.

apiVersion: serving.llm-d.ai/v1alpha1
kind: LLMInferenceService
metadata:
  name: llama-3-8b
  namespace: ai-inference
  labels:
    app.kubernetes.io/part-of: inference-platform
    app.kubernetes.io/managed-by: redhat-ai-operator
spec:
  # --- Model Source ---
  model:
    name: meta-llama/Llama-3.1-8B-Instruct   # HuggingFace model ID or custom name
    source:
      uri: pvc://model-store/llama-3-8b        # PVC-backed model storage
    maxModelLen: 8192                          # Max sequence length (context window)
    dtype: float16                               # Model precision: float16, bfloat16, auto

  # --- Serving Runtime ---
  serving:
    runtime: vllm                                # Inference engine: vllm is the default
    image: registry.redhat.io/redhat-ai/vllm-rhel9:0.8.4-12  # Version-pinned image
    replicas: 2                                  # Number of inference pod replicas
    resources:
      gpu: nvidia.com/gpu                        # GPU resource name from device plugin
      gpuCount: 1                                # GPUs per replica (1 for 8B model)
      memory: 32Gi                                # CPU memory per replica
      cpu: 8                                      # CPU cores per replica

  # --- Routing ---
  routing:
    endpoint: EPP                                # Endpoint Picker for intelligent routing
    strategy: round-robin                       # Basic round-robin for simple workloads

  # --- Autoscaling ---
  autoscaling:
    minReplicas: 1                               # Minimum replicas (1 = never scale to zero)
    maxReplicas: 4                               # Maximum replicas ceiling
    targetUtilization: 80                       # Scale when utilization exceeds 80%

Every field has operational significance. The model.source.uri supports multiple schemes: pvc:// for models stored on PersistentVolumeClaims, s3:// for S3-compatible object storage, oci:// for models packed as OCI container images, and hf:// for direct download from HuggingFace Hub (not recommended for production due to startup latency and network dependency). The maxModelLen directly controls KV-cache memory allocation -- setting this higher than necessary wastes GPU memory, while setting it too low truncates long conversations.

The serving.image field pins the exact vLLM container image version. In production, never use :latest tags. The image registry.redhat.io/redhat-ai/vllm-rhel9:0.8.4-12 is built from audited source, signed with Red Hat's GPG key, and scanned for CVEs. The -12 suffix denotes the build iteration, which includes security patches applied on top of the vLLM 0.8.4 release.

The serving.resources.gpuCount determines tensor parallelism. For an 8B parameter model in float16 (roughly 16 GB of weights), a single GPU with 80 GB of VRAM is more than sufficient. The remaining GPU memory becomes available for KV-cache, which directly determines how many concurrent requests the replica can handle.

Disaggregated Serving Deployment

Disaggregated serving splits the inference pipeline into two distinct phases: prefill (processing the input prompt and building the initial KV-cache) and decode (generating output tokens autoregressively). These two phases have fundamentally different compute profiles. Prefill is compute-bound and benefits from high FLOPS. Decode is memory-bandwidth-bound and benefits from high memory throughput. By running them on separate replica pools, each can be optimized independently.

apiVersion: serving.llm-d.ai/v1alpha1
kind: LLMInferenceService
metadata:
  name: llama-3-70b-disagg
  namespace: ai-inference
  labels:
    app.kubernetes.io/part-of: inference-platform
spec:
  model:
    name: meta-llama/Llama-3.1-70B-Instruct
    source:
      uri: pvc://model-store/llama-3-70b
    maxModelLen: 32768
    dtype: bfloat16

  serving:
    runtime: vllm
    image: registry.redhat.io/redhat-ai/vllm-rhel9:0.8.4-12
    serviceAccountName: inference-sa            # Dedicated service account
    resources:
      gpu: nvidia.com/gpu
      gpuCount: 4                                # 4-way tensor parallelism per replica
      gpuType: H100-SXM                            # Require SXM form factor for NVLink
      memory: 128Gi
      cpu: 32

    # --- Security Context ---
    securityContext:
      runAsNonRoot: true                          # Never run as root
      readOnlyRootFilesystem: true                # Immutable container filesystem
      allowPrivilegeEscalation: false            # Block privilege escalation
      seccompProfile:
        type: RuntimeDefault

    # --- Disaggregated Serving Config ---
    disaggregated:
      enabled: true                               # Enable prefill/decode splitting
      prefillReplicas: 2                          # Dedicated prefill workers
      decodeReplicas: 4                           # Dedicated decode workers
      kvTransferMethod: nccl                     # KV-cache transfer: nccl, tcp, rdma

  # --- PodDisruptionBudget ---
  pdb:
    minAvailable: 4                               # At least 4 replicas during disruptions

  routing:
    endpoint: EPP
    strategy: kv-cache-aware                    # Route based on KV-cache state
    prefixCacheAware: true                      # Reuse prefix cache across requests

  autoscaling:
    minReplicas: 2
    maxReplicas: 12
    targetUtilization: 70
    scaleUpStabilization: 60                   # Seconds to wait before scaling up
    scaleDownStabilization: 300                # Seconds to wait before scaling down
    metric: queue-depth                          # Scale on queue depth, not CPU

The production-hardening fields in this YAML are critical for enterprise deployments. The securityContext block enforces the principle of least privilege: runAsNonRoot: true prevents the container from running as UID 0, readOnlyRootFilesystem: true prevents file writes to the container overlay (model weights are mounted read-only from the PVC), and allowPrivilegeEscalation: false blocks any capability escalation. The seccompProfile: RuntimeDefault applies the CRI-O default seccomp profile, blocking dangerous syscalls.

The pdb.minAvailable: 4 ensures that during voluntary disruptions (node maintenance, driver upgrades, rolling updates), at least 4 replicas remain serving traffic. This prevents the cluster administrator from accidentally evicting all inference pods during a maintenance window.

The ratio of prefill to decode replicas matters. Prefill is computationally expensive but fast per-request (it processes the entire prompt in one forward pass). Decode is cheaper per-step but handles many concurrent, long-running generation streams. A common starting ratio is 1:2 (prefill:decode) and is adjusted based on observed workload characteristics. The kvTransferMethod controls how KV-cache state moves from prefill to decode workers -- nccl uses NVIDIA Collective Communication Library for high-bandwidth GPU-to-GPU transfer, while tcp falls back to standard networking when GPUs are not on the same interconnect fabric.

The autoscaling metric field is critical. Default Kubernetes HPA scales on CPU utilization, which is nearly meaningless for GPU inference workloads. Setting metric: queue-depth tells the autoscaler to watch the number of pending requests, which directly correlates with user-visible latency. The stabilization windows prevent thrashing: scaleUpStabilization: 60 means the system must see sustained high demand for 60 seconds before adding replicas, and scaleDownStabilization: 300 provides a 5-minute cooldown before removing them.

Multi-LoRA Deployment

LoRA (Low-Rank Adaptation) adapters allow a single base model to serve multiple fine-tuned variants without loading separate copies of the full model weights. This is extraordinarily GPU-efficient: rather than allocating 140 GB of VRAM per fine-tuned model, you load the base model once and swap lightweight adapters (typically 50-200 MB each) per request.

apiVersion: serving.llm-d.ai/v1alpha1
kind: LLMInferenceService
metadata:
  name: llama-3-70b-multi-lora
  namespace: ai-inference
spec:
  model:
    name: meta-llama/Llama-3.1-70B-Instruct
    source:
      uri: pvc://model-store/llama-3-70b
    maxModelLen: 16384
    dtype: bfloat16
    loraAdapters:                                 # List of LoRA adapters to load
      - name: customer-support-v2
        source: pvc://lora-store/customer-support-v2
        maxLoraRank: 64
      - name: code-assist-v1
        source: pvc://lora-store/code-assist-v1
        maxLoraRank: 32
      - name: legal-review-v3
        source: pvc://lora-store/legal-review-v3
        maxLoraRank: 64

  serving:
    runtime: vllm
    image: registry.redhat.io/redhat-ai/vllm-rhel9:0.8.4-12
    serviceAccountName: inference-sa
    replicas: 4
    resources:
      gpu: nvidia.com/gpu
      gpuCount: 4
      gpuType: H100-SXM
      memory: 128Gi
      cpu: 32
    securityContext:
      runAsNonRoot: true
      readOnlyRootFilesystem: true
      allowPrivilegeEscalation: false

  pdb:
    minAvailable: 2

  routing:
    endpoint: EPP
    strategy: lora-affinity                     # Route by adapter affinity

  autoscaling:
    minReplicas: 2
    maxReplicas: 8
    targetUtilization: 75

The routing.strategy: lora-affinity is key. When a request arrives specifying the customer-support-v2 adapter, the EPP routes it to a replica that already has that adapter loaded in GPU memory, avoiding adapter swap overhead. If no replica has the adapter warm, the EPP selects the replica with the lowest current load and triggers an adapter load. The maxLoraRank controls the adapter size -- higher ranks enable more expressive fine-tuning but consume more GPU memory.

How LLMInferenceService Extends KServe Under the Hood #

To fully appreciate the LLMInferenceService CRD, you need to understand the KServe architecture it builds on and the ways it diverges from it. KServe was designed as a general-purpose model serving framework. LLM inference breaks many of its original assumptions, and llm-d's CRDs are the result of rethinking those assumptions from first principles.

KServe's Architecture: Predictor, Transformer, Explainer

KServe's InferenceService CRD is organized around three components. The Predictor is the core model serving container -- it loads the model, handles inference requests, and returns predictions. The Transformer sits in front of the predictor and handles pre-processing (tokenization, feature extraction, input validation) and post-processing (detokenization, response formatting). The Explainer is an optional component that provides model explainability (SHAP values, attention maps, feature importance).

This three-component model works well for traditional ML workloads -- image classification, tabular prediction, NLP classification -- where the inference pipeline is a clean request-response cycle. But LLM inference shatters this model in several ways. First, LLM inference is stateful: the KV-cache persists across the lifetime of a generation, and routing decisions must account for cache locality. Second, LLM inference is streaming: tokens are produced incrementally over seconds or minutes, not in a single response. Third, LLM inference is heterogeneous: prefill and decode have fundamentally different compute profiles and benefit from different hardware configurations. KServe's InferenceService has no concept of any of these.

How LLMInferenceService Maps to (and Extends) InferenceService

The LLMInferenceService CRD inherits KServe's declarative philosophy -- you describe what you want, and a controller reconciles the real world to match -- but replaces the generic Predictor/Transformer/Explainer model with LLM-specific abstractions. Internally, the controller still generates KServe-compatible resources where possible (for example, it can create an InferenceService as a child resource for compatibility with KServe's monitoring ecosystem), but the top-level API is purpose-built for LLMs.

The CRD spec is organized into six top-level field groups, each governing a distinct operational concern:

• model -- Defines what to serve: the model identity, source location (PVC, S3, OCI, HuggingFace), precision (dtype), context window (maxModelLen), quantization method, and LoRA adapter configuration. This replaces KServe's generic predictor.model with fields that understand LLM-specific concerns like adapter rank and KV-cache memory allocation.
• serving -- Defines how to serve: the runtime engine (vLLM), container image, replica count, resource requests (GPU type, GPU count, CPU, memory), security context, disaggregated serving configuration (prefill/decode split, KV transfer method), and pod template overrides. This is the most complex field group because it encapsulates both the inference engine configuration and the Kubernetes workload configuration.
• routing -- Defines how requests reach the right replica: the endpoint picker type (EPP), routing strategy (round-robin, kv-cache-aware, lora-affinity), and prefix cache awareness. This entire field group has no equivalent in KServe, which defers routing to Kubernetes Services and Istio VirtualServices.
• autoscaling -- Defines when to scale: minimum and maximum replica counts, target utilization, scale-up and scale-down stabilization windows, and the metric that drives scaling decisions (queue-depth, kv-cache-utilization, or a custom Prometheus metric). While KServe supports Knative autoscaling, it does not understand inference-specific metrics.
• pdb -- Defines disruption tolerance: the minimum number of replicas that must remain available during voluntary disruptions (node drains, rolling updates, operator upgrades). This maps directly to a Kubernetes PodDisruptionBudget, but is elevated to a top-level field because disruption management is critical for inference SLAs.
• observability -- Defines what to monitor: metric export configuration, ServiceMonitor labels, custom metric endpoints. When present, the controller creates ServiceMonitor resources that integrate with OpenShift's Prometheus instance. When absent, the controller uses sensible defaults.

Resource Dependency Graph

When you apply an LLMInferenceService resource, the controller creates a tree of downstream Kubernetes resources. Understanding this tree is essential for debugging and for understanding the blast radius of changes:

• LLMInferenceService (the parent resource you create)
  • Deployment (or StatefulSet for disaggregated serving) -- runs the vLLM inference pods
  • Service -- internal ClusterIP service for pod-to-pod communication
  • ServiceMonitor -- tells Prometheus how to scrape inference metrics from the pods
  • HorizontalPodAutoscaler -- scales replicas based on the configured metric and thresholds
  • PodDisruptionBudget -- protects replicas during voluntary disruptions
  • Route (OpenShift-specific) -- exposes the inference endpoint with TLS termination
  • EPP Deployment + Service -- the Endpoint Picker that routes requests intelligently
  • ConfigMap -- vLLM engine configuration derived from CRD fields

All of these resources are owned by the LLMInferenceService via Kubernetes owner references. If you delete the parent, all children are garbage collected. If you modify the parent, the controller reconciles all children to match. This ownership model means you should never manually edit the child resources -- your changes will be overwritten on the next reconciliation loop. If you need to customize a child resource (for example, adding a sidecar container to the inference pod), use the serving.podTemplate override field in the CRD.

InferencePool and InferenceGateway CRDs

Beyond LLMInferenceService, the llm-d project defines two additional CRDs that operate at a higher level of abstraction: InferencePool and InferenceGateway.

An InferencePool groups multiple LLMInferenceService instances that share common routing infrastructure. The pool-level EPP handles cross-service routing decisions -- for example, routing a request to the service with the most available KV-cache capacity, regardless of which specific model or LoRA adapter it serves. This is useful for organizations running multiple models behind a unified API gateway.

apiVersion: serving.llm-d.ai/v1alpha1
kind: InferencePool
metadata:
  name: production-pool
  namespace: ai-inference
spec:
  # --- EPP Configuration ---
  endpointPicker:
    image: registry.redhat.io/redhat-ai/epp-rhel9:3.4-8
    replicas: 3                                    # HA EPP for the pool
    resources:
      cpu: 4
      memory: 8Gi
    scoringStrategy:
      kvCacheWeight: 0.4                            # 40% weight on KV-cache availability
      queueDepthWeight: 0.3                        # 30% weight on queue depth
      loraAffinityWeight: 0.2                      # 20% weight on LoRA adapter locality
      prefixCacheWeight: 0.1                       # 10% weight on prefix cache hits

  # --- Member Services ---
  selector:
    matchLabels:
      pool: production                              # Select LLMInferenceServices with this label

  # --- Health Policy ---
  healthPolicy:
    unhealthyThreshold: 3                          # Mark backend unhealthy after 3 failed checks
    healthCheckInterval: 10s                       # Health check every 10 seconds

An InferenceGateway sits at the edge of the inference infrastructure, providing API gateway capabilities -- rate limiting, authentication, request validation, and traffic management -- before requests reach the InferencePool or individual LLMInferenceService endpoints.

apiVersion: serving.llm-d.ai/v1alpha1
kind: InferenceGateway
metadata:
  name: production-gateway
  namespace: ai-inference
spec:
  # --- Target Pool ---
  poolRef:
    name: production-pool

  # --- Rate Limiting ---
  rateLimiting:
    enabled: true
    requestsPerMinute: 600                        # Global rate limit
    tokensPerMinute: 100000                        # Token-based rate limit
    perClientLimits:
      enabled: true
      headerKey: X-API-Key                        # Identify clients by API key header
      requestsPerMinute: 60                       # Per-client limit

  # --- Authentication ---
  authentication:
    type: bearer                                   # Bearer token authentication
    tokenValidation:
      issuer: https://sso.example.com/realms/ai    # OIDC issuer for JWT validation
      audience: inference-api

  # --- TLS ---
  tls:
    mode: terminate
    certificateRef:
      name: inference-tls-cert
      namespace: ai-inference

  # --- Request Validation ---
  requestValidation:
    maxInputTokens: 32768                         # Reject requests exceeding context window
    maxOutputTokens: 4096                          # Cap maximum generation length
    blockedModels: []                                # Optionally block specific model names

The relationship between these three CRDs forms a hierarchy: InferenceGateway handles edge concerns (auth, rate limiting, TLS), InferencePool handles fleet-level routing (load balancing across services), and LLMInferenceService handles individual model deployment (replicas, GPU resources, scaling). You can use any level independently -- a single LLMInferenceService creates its own Route and EPP, suitable for simple deployments -- but the full hierarchy provides the operational controls that multi-team, multi-model enterprise deployments require.

OpenShift GPU Operator Deep Dive #

Every distributed inference deployment is, at its foundation, a GPU orchestration problem. The OpenShift GPU Operator manages the complete lifecycle of NVIDIA GPU software on every node in the cluster. It eliminates the historically painful process of manually installing drivers, CUDA toolkits, and container runtime hooks on bare-metal GPU nodes.

ClusterPolicy CRD

The GPU Operator is configured through the ClusterPolicy CRD. This single resource defines the desired state for GPU software across the entire cluster:

apiVersion: nvidia.com/v1
kind: ClusterPolicy
metadata:
  name: gpu-cluster-policy
spec:
  operator:
    defaultRuntime: crio                        # OpenShift uses CRI-O, not containerd
  driver:
    enabled: true
    version: 550.127.08                         # NVIDIA driver version, pinned
    upgradePolicy:
      autoUpgrade: true                           # Enable rolling driver upgrades
      maxParallelUpgrades: 1                     # One node at a time
      maxUnavailable: 25%                        # Max nodes unavailable during upgrade
      waitForCompletion:
        timeoutSeconds: 0                        # 0 = no timeout, wait for completion
      podDeletion:
        force: false                               # Don't force-delete running pods
        timeoutSeconds: 300                       # Wait 5 min for graceful termination
        deleteEmptyDir: false
  toolkit:
    enabled: true                                  # NVIDIA Container Toolkit for GPU passthrough
  devicePlugin:
    enabled: true                                  # Kubernetes Device Plugin for nvidia.com/gpu
  dcgmExporter:
    enabled: true                                  # DCGM metrics exporter for Prometheus
  gfd:
    enabled: true                                  # GPU Feature Discovery for node labels
  mig:
    strategy: single                              # MIG strategy: single, mixed, or none

How the GPU Operator Works

The GPU Operator deploys as a DaemonSet-based architecture with five core components. When a new GPU node joins the cluster, the operator detects it and deploys a stack of components in a specific order:

1. NVIDIA Driver Container -- A privileged container that compiles and installs the NVIDIA kernel driver on the host OS. This runs as a DaemonSet pod on every GPU node. The driver container approach means the host OS never needs manual driver installation, and driver versions are managed declaratively through the ClusterPolicy.
2. NVIDIA Container Toolkit -- Configures the CRI-O container runtime (OpenShift's container runtime) to pass GPU devices into containers. It installs the nvidia-container-runtime-hook that intercepts container creation calls and injects the necessary device files and libraries.
3. Kubernetes Device Plugin -- Registers nvidia.com/gpu as a schedulable resource with the kubelet. When a pod requests nvidia.com/gpu: 4, the device plugin allocates four specific GPU devices and makes them available inside the container.
4. GPU Feature Discovery (GFD) -- Inspects the physical GPU hardware and applies Kubernetes node labels describing the GPU model, memory, driver version, CUDA version, and hardware capabilities. These labels are critical for scheduling.
5. DCGM Exporter -- Exposes GPU metrics (utilization, temperature, memory usage, ECC errors, power draw) as Prometheus metrics, scraped automatically by OpenShift's monitoring stack.

Driver Upgrades Without Node Drain

The upgradePolicy in the ClusterPolicy enables rolling driver upgrades without manual node draining. When a new driver version is set, the operator proceeds node-by-node: it cordons the node to prevent new pod scheduling, gracefully evicts GPU workloads (respecting PodDisruptionBudgets), updates the driver container to the new version, validates that the driver loads correctly, and then uncordons the node. The maxParallelUpgrades: 1 ensures only one node is being upgraded at a time, maintaining cluster capacity during the process. The maxUnavailable: 25% acts as a circuit breaker: if more than 25% of GPU nodes are unhealthy, the upgrade pauses automatically.

"Automated driver upgrades alone saved us 200 engineer-hours per quarter across our 40-node GPU cluster."

-- SRE Manager, Autonomous Vehicle Company

Node Feature Discovery and GPU Topology-Aware Scheduling #

Getting GPU workloads to run is table stakes. Getting them to run on the right GPUs is where performance is won or lost. Node Feature Discovery (NFD) is the operator that bridges the gap between physical hardware topology and Kubernetes scheduling decisions.

GPU Node Labels

When NFD and GPU Feature Discovery run on a node with NVIDIA H100 SXM GPUs, they generate labels like the following:

$ oc get node gpu-node-01 -o json | jq '.metadata.labels | to_entries[] | select(.key | startswith("nvidia.com") or startswith("feature.node.kubernetes.io/pci"))'

nvidia.com/gpu.product=NVIDIA-H100-SXM-80GB
nvidia.com/gpu.memory=81920                      # GPU memory in MiB
nvidia.com/gpu.count=8                            # Total GPUs on node
nvidia.com/gpu.family=hopper                     # GPU architecture family
nvidia.com/gpu.compute.major=9                   # CUDA compute capability major
nvidia.com/gpu.compute.minor=0                   # CUDA compute capability minor
nvidia.com/cuda.driver.major=550                 # Driver version major
nvidia.com/cuda.runtime.major=12                # CUDA runtime major
nvidia.com/gpu.deploy.driver=true
nvidia.com/gpu.deploy.device-plugin=true
nvidia.com/gpu.deploy.dcgm-exporter=true
feature.node.kubernetes.io/pci-10de.present=true   # NVIDIA PCI vendor ID
feature.node.kubernetes.io/cpu-model.vendor_id=Intel
feature.node.kubernetes.io/system-os_release.ID=rhcos

Topology-Aware Scheduling for Tensor Parallelism

When an LLMInferenceService specifies gpuCount: 4 for a 70B model, those four GPUs must be on the same NVLink domain for efficient tensor-parallel communication. NVLink provides 900 GB/s bidirectional bandwidth between GPUs in the same domain, compared to 64 GB/s over PCIe Gen5. Running tensor parallelism over PCIe instead of NVLink can degrade throughput by 3-5x.

The Kubernetes scheduler uses the GPU node labels and the NVIDIA device plugin's topology awareness to ensure that when a pod requests four GPUs, it receives four GPUs that share an NVLink mesh. On an 8-GPU H100 SXM node, GPUs are typically arranged in two NVLink domains of four GPUs each, corresponding to the two CPU sockets and their NUMA nodes.

For workloads that need explicit control over topology spread, the CRD supports standard Kubernetes topologySpreadConstraints and nodeAffinity rules. For example, to ensure prefill and decode replicas do not compete for GPUs on the same node:

spec:
  serving:
    podTemplate:
      spec:
        topologySpreadConstraints:
          - maxSkew: 1
            topologyKey: kubernetes.io/hostname
            whenUnsatisfiable: DoNotSchedule
            labelSelector:
              matchLabels:
                app: llama-3-70b-disagg
        nodeSelector:
          nvidia.com/gpu.product: NVIDIA-H100-SXM-80GB
          nvidia.com/gpu.count: "8"                # Only schedule on 8-GPU nodes

The nodeSelector restricts scheduling to nodes with the exact GPU model and count. The topologySpreadConstraints with maxSkew: 1 ensures replicas are evenly distributed across physical nodes. The DoNotSchedule policy means the pod stays Pending rather than landing on a suboptimal node -- a deliberate choice, because running a tensor-parallel workload on mismatched GPU topology produces worse results than not running it at all.

Monitoring Integration: Prometheus, Grafana, and Alerting #

LLM inference generates a set of operational signals that generic application monitoring does not capture. Standard dashboards that show CPU, memory, and network utilization tell you almost nothing about whether your inference service is healthy. What matters is inference-specific telemetry: latency distributions, cache utilization, queue dynamics, and GPU memory pressure.

The llm-d inference stack (built on vLLM) exposes a rich set of Prometheus metrics. OpenShift's built-in Prometheus instance scrapes these automatically through ServiceMonitor resources that the LLMInferenceService controller creates. No manual scrape configuration is required.

Key PromQL Queries

Time-to-First-Token P99

TTFT is the single most important metric for user-perceived responsiveness. It measures the time from when a request arrives to when the first output token is generated. A P99 above 500ms usually indicates either KV-cache saturation or insufficient prefill capacity.

histogram_quantile(0.99,
  sum(rate(vllm:time_to_first_token_seconds_bucket{
    namespace="ai-inference",
    model_name="llama-3-70b"
  }[5m])) by (le)
)

Inter-Token Latency P95

ITL measures the gap between successive output tokens during streaming decode. Spikes here cause visible "stuttering" in real-time streaming responses. Sustained ITL above 100ms is perceptible to users.

histogram_quantile(0.95,
  sum(rate(vllm:inter_token_latency_seconds_bucket{
    namespace="ai-inference",
    model_name="llama-3-70b"
  }[5m])) by (le)
)

KV-Cache Utilization per Replica

KV-cache utilization is the leading indicator for capacity exhaustion. When a replica's KV-cache is full, new requests must wait for existing sequences to complete, directly increasing queue depth and TTFT.

vllm:gpu_cache_usage_perc{
  namespace="ai-inference",
  model_name="llama-3-70b"
} * 100

Request Queue Depth

Queue depth is the number of requests waiting for an inference slot. This metric should trigger scaling decisions. A sustained queue depth greater than zero means requests are being delayed.

sum(vllm:num_requests_waiting{
  namespace="ai-inference",
  model_name="llama-3-70b"
}) by (pod)

GPU Memory Utilization

GPU memory utilization from DCGM, correlated with inference metrics, reveals whether the system is memory-bound (high memory, low compute utilization) or compute-bound (high compute, moderate memory).

DCGM_FI_DEV_FB_USED{namespace="ai-inference"}
/ on(pod, gpu)
DCGM_FI_DEV_FB_FREE{namespace="ai-inference"}
* 100

Alerting Rules

Alerting on inference workloads requires inference-specific thresholds. Generic "CPU above 90%" alerts are useless here. The following PrometheusRule resource defines alerts that map to actual user-impact scenarios:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: llm-inference-alerts
  namespace: ai-inference
spec:
  groups:
    - name: llm-inference.rules
      rules:
        # TTFT SLA breach: P99 > 500ms for 5 minutes
        - alert: HighTimeToFirstToken
          expr: |
            histogram_quantile(0.99,
              sum(rate(vllm:time_to_first_token_seconds_bucket[5m])) by (le, model_name)
            ) > 0.5
          for: 5m
          labels:
            severity: critical
          annotations:
            summary: "TTFT P99 exceeds 500ms for {{ $labels.model_name }}"
            description: "Time-to-first-token P99 is {{ $value | humanize }}s. Check KV-cache utilization and prefill replica count."

        # KV-cache saturation: any replica above 90%
        - alert: KVCacheSaturation
          expr: vllm:gpu_cache_usage_perc > 0.9
          for: 3m
          labels:
            severity: warning
          annotations:
            summary: "KV-cache above 90% on {{ $labels.pod }}"
            description: "KV-cache utilization is {{ $value | humanizePercentage }}. New requests will queue. Consider scaling replicas or reducing maxModelLen."

        # Queue depth: requests waiting for more than 2 minutes
        - alert: InferenceQueueBacklog
          expr: sum(vllm:num_requests_waiting) by (model_name) > 10
          for: 2m
          labels:
            severity: warning
          annotations:
            summary: "Inference queue backlog for {{ $labels.model_name }}"
            description: "{{ $value }} requests queued. Autoscaler should be increasing replicas. If this persists, check maxReplicas ceiling."

        # GPU memory exhaustion: DCGM reports > 95% FB usage
        - alert: GPUMemoryExhaustion
          expr: |
            DCGM_FI_DEV_FB_USED / (DCGM_FI_DEV_FB_USED + DCGM_FI_DEV_FB_FREE) > 0.95
          for: 5m
          labels:
            severity: critical
          annotations:
            summary: "GPU memory above 95% on {{ $labels.pod }}"
            description: "GPU {{ $labels.gpu }} is near memory exhaustion. Risk of OOM. Reduce maxModelLen, batch size, or add replicas."

Grafana Dashboard Snippet

Red Hat AI ships with pre-built Grafana dashboards for inference observability. The following JSON snippet shows the panel configuration for a TTFT P99 time-series panel that you can import into any Grafana instance:

{
  "title": "Time-to-First-Token P99",
  "type": "timeseries",
  "datasource": { "type": "prometheus", "uid": "openshift-monitoring" },
  "targets": [
    {
      "expr": "histogram_quantile(0.99, sum(rate(vllm:time_to_first_token_seconds_bucket{namespace=\"ai-inference\"}[5m])) by (le, model_name))",
      "legendFormat": "{{ model_name }}",
      "refId": "A"
    }
  ],
  "fieldConfig": {
    "defaults": {
      "unit": "s",
      "thresholds": {
        "steps": [
          { "value": 0, "color": "green" },
          { "value": 0.3, "color": "yellow" },
          { "value": 0.5, "color": "red" }
        ]
      }
    }
  },
  "options": {
    "tooltip": { "mode": "multi" },
    "legend": { "displayMode": "table", "placement": "bottom" }
  }
}

"The pre-built Grafana dashboards caught a KV-cache memory leak that would have caused a production outage."

-- ML Platform Lead, E-Commerce

Step-by-Step Deployment Walkthrough #

This section traces the concrete steps to go from an OpenShift cluster with GPU nodes to a production inference endpoint serving traffic. Every command is shown with its expected output.

Step 1: Install the Red Hat AI Operator

The Red Hat AI operator is installed from OperatorHub. This deploys the controllers that understand the LLMInferenceService CRD and manage the inference lifecycle.

apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: redhat-ai-operator
  namespace: openshift-operators
spec:
  channel: stable-3.4                             # Red Hat AI 3.4 channel
  installPlanApproval: Automatic
  name: redhat-ai-operator
  source: redhat-operators
  sourceNamespace: openshift-marketplace

# Create the operator namespace
$ oc create namespace redhat-ai-operator

# Apply the operator subscription
$ oc apply -f redhat-ai-subscription.yaml

# Verify the operator pod is running
$ oc get pods -n openshift-operators -l app=redhat-ai-operator
NAME                                    READY   STATUS    RESTARTS   AGE
redhat-ai-operator-6d8f9b7c4d-x2k9m    1/1     Running   0          2m15s

Step 2: Configure the GPU Operator

If the GPU Operator is not already installed, install it and apply the ClusterPolicy:

# Install the GPU Operator (if not already present)
$ oc apply -f - <<EOF
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: gpu-operator-certified
  namespace: nvidia-gpu-operator
spec:
  channel: v24.9
  installPlanApproval: Automatic
  name: gpu-operator-certified
  source: certified-operators
  sourceNamespace: openshift-marketplace
EOF

# Apply ClusterPolicy (see GPU Operator section above for full YAML)
$ oc apply -f cluster-policy.yaml

# Verify GPU nodes are discovered
$ oc get nodes -l nvidia.com/gpu.present=true
NAME           STATUS   ROLES    AGE   VERSION
gpu-node-01    Ready    worker   45d   v1.29.9+openshift.1
gpu-node-02    Ready    worker   45d   v1.29.9+openshift.1
gpu-node-03    Ready    worker   45d   v1.29.9+openshift.1

# Confirm GPUs are allocatable
$ oc describe node gpu-node-01 | grep -A 4 "Allocatable:"
Allocatable:
  cpu:                128
  memory:             1056784Mi
  nvidia.com/gpu:     8
  pods:               250

Step 3: Create Namespace and Model Storage

# Create the inference namespace
$ oc new-project ai-inference

# Create a PVC for model weights (backed by high-throughput storage)
$ oc apply -f - <<EOF
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: model-store
  namespace: ai-inference
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: ocs-storagecluster-cephfs
  resources:
    requests:
      storage: 500Gi
EOF

# Download model weights into the PVC using a Job
$ oc apply -f - <<EOF
apiVersion: batch/v1
kind: Job
metadata:
  name: download-llama-3-70b
  namespace: ai-inference
spec:
  template:
    spec:
      containers:
        - name: downloader
          image: registry.redhat.io/redhat-ai/model-downloader-rhel9:3.4-2
          env:
            - name: MODEL_ID
              value: meta-llama/Llama-3.1-70B-Instruct
            - name: HF_TOKEN
              valueFrom:
                secretKeyRef:
                  name: hf-credentials
                  key: token
          volumeMounts:
            - name: model-store
              mountPath: /models
              subPath: llama-3-70b
      volumes:
        - name: model-store
          persistentVolumeClaim:
            claimName: model-store
      restartPolicy: Never
EOF

# Monitor download progress
$ oc logs -f job/download-llama-3-70b -n ai-inference
Downloading meta-llama/Llama-3.1-70B-Instruct...
[========================================] 100%  140.4 GB / 140.4 GB
Model downloaded successfully to /models/llama-3-70b

Step 4: Apply the LLMInferenceService

# Apply the LLMInferenceService (using the disaggregated YAML from earlier)
$ oc apply -f disaggregated-inference.yaml
llminferenceservice.serving.llm-d.ai/llama-3-70b-disagg created

# Watch the controller reconcile
$ oc get llminferenceservice -n ai-inference -w
NAME                  READY   REPLICAS   ROUTING   AGE
llama-3-70b-disagg    False   0/6        EPP       5s
llama-3-70b-disagg    False   2/6        EPP       45s
llama-3-70b-disagg    False   4/6        EPP       1m30s
llama-3-70b-disagg    True    6/6        EPP       3m12s

Step 5: Verify Pods Are Running

$ oc get pods -n ai-inference -l llm-d.ai/service=llama-3-70b-disagg
NAME                                           READY   STATUS    RESTARTS   AGE
llama-3-70b-disagg-prefill-0                   1/1     Running   0          4m
llama-3-70b-disagg-prefill-1                   1/1     Running   0          4m
llama-3-70b-disagg-decode-0                    1/1     Running   0          4m
llama-3-70b-disagg-decode-1                    1/1     Running   0          4m
llama-3-70b-disagg-decode-2                    1/1     Running   0          4m
llama-3-70b-disagg-decode-3                    1/1     Running   0          4m
llama-3-70b-disagg-epp-7f8d4b6c9-m2kpq        1/1     Running   0          4m

# Verify GPU allocation
$ oc describe pod llama-3-70b-disagg-prefill-0 -n ai-inference | grep -A 2 "Limits:"
    Limits:
      nvidia.com/gpu:  4
      memory:          128Gi

Step 6: Test the Endpoint

# Get the Route URL
$ INFERENCE_URL=$(oc get route llama-3-70b-disagg -n ai-inference -o jsonpath='{.spec.host}')

# Send a test request
$ curl -s https://${INFERENCE_URL}/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-3.1-70B-Instruct",
    "messages": [
      {"role": "user", "content": "Explain Kubernetes in one paragraph."}
    ],
    "max_tokens": 256,
    "temperature": 0.7
  }' | jq '.choices[0].message.content'

"Kubernetes is an open-source container orchestration platform that automates
the deployment, scaling, and management of containerized applications..."

# Verify streaming works
$ curl -N https://${INFERENCE_URL}/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-3.1-70B-Instruct",
    "messages": [{"role": "user", "content": "Hello!"}],
    "max_tokens": 50,
    "stream": true
  }'
data: {"choices":[{"delta":{"content":"Hello"},"index":0}],...}
data: {"choices":[{"delta":{"content":"!"},"index":0}],...}
data: {"choices":[{"delta":{"content":" How"},"index":0}],...}
...

Step 7: Check Metrics

# Port-forward to a vLLM pod's metrics endpoint
$ oc port-forward llama-3-70b-disagg-decode-0 -n ai-inference 8080:8080 &

# Query metrics directly
$ curl -s localhost:8080/metrics | grep vllm:num_requests
vllm:num_requests_running{model_name="llama-3-70b"} 3
vllm:num_requests_waiting{model_name="llama-3-70b"} 0

$ curl -s localhost:8080/metrics | grep gpu_cache_usage
vllm:gpu_cache_usage_perc{model_name="llama-3-70b"} 0.234

# Verify ServiceMonitor exists (auto-created by the controller)
$ oc get servicemonitor -n ai-inference
NAME                              AGE
llama-3-70b-disagg-metrics        5m

Troubleshooting Guide #

Even with operator-managed deployments, things go wrong. This section covers the six most common failure modes, their root causes, and the steps to resolve them. Each is assigned a severity level consistent with Red Hat's support classification (see the Support SLAs section below). These are drawn from real-world production deployments.

Support SLAs and Incident Response #

For enterprise architects, the support contract is often the deciding factor between upstream and commercial. This section details the specific response commitments that come with a Red Hat AI subscription, mapped to LLM inference failure scenarios.

Support Tiers

Severity Classification for LLM Inference

Red Hat's standard severity definitions apply, but the following table maps them specifically to LLM inference failure modes:

Escalation Procedures

When a Sev-1 incident occurs in production LLM inference, the escalation path is:

1. Initial contact -- Call the Red Hat support hotline (not the web portal) for Sev-1 issues. Phone cases are routed to engineers with GPU and inference expertise.
2. Data collection -- Run oc adm must-gather --image=registry.redhat.io/redhat-ai/must-gather-rhel9:3.4 to collect a comprehensive diagnostic bundle. This captures GPU operator state, inference pod logs, DCGM metrics, and cluster configuration.
3. TAM engagement -- Premium subscribers have a designated Technical Account Manager who coordinates between the customer's team and Red Hat engineering. The TAM ensures the right specialist is engaged for inference-specific issues.
4. Engineering escalation -- If the support engineer cannot resolve the issue, it escalates to Red Hat's AI Platform engineering team, who have direct access to the llm-d upstream codebase and vLLM internals.
5. Hot fix delivery -- For confirmed product defects in Sev-1 scenarios, Red Hat can deliver a hot fix as a patched container image within hours, before the fix appears in the next scheduled errata release.

CVE Response and Security Lifecycle #

LLM inference infrastructure processes potentially sensitive data (user prompts, generated content, fine-tuning datasets) and runs GPU-privileged workloads. The security posture of this stack is not optional -- it is a hard requirement for any regulated industry.

Red Hat Product Security Process

Red Hat Product Security maintains a dedicated team that monitors CVE databases, upstream security advisories, and community vulnerability reports for every component in the Red Hat AI stack. When a vulnerability is identified:

1. Triage -- The vulnerability is assessed for impact on Red Hat AI specifically. A CVE in a Python library used by vLLM may not affect the Red Hat build if the vulnerable code path is not reachable.
2. CVSS scoring -- Red Hat assigns its own CVSS score that reflects the actual risk in the context of the OpenShift deployment model (e.g., a network-exploitable vulnerability has lower impact if the inference endpoint is behind an authenticated Route).
3. Patch development -- For critical and important CVEs, patches are backported to the supported release branch without requiring a version upgrade.
4. Errata publication -- Patched container images are published as RHSA (Red Hat Security Advisory) errata, with full documentation of the vulnerability, impact, and remediation steps.

Container Image Signing

Every container image published to registry.redhat.io is signed with Red Hat's GPG key. OpenShift can be configured to reject unsigned images, ensuring that only verified images run in the cluster:

# Verify the vLLM image signature
$ skopeo inspect --tls-verify=true docker://registry.redhat.io/redhat-ai/vllm-rhel9:0.8.4-12 | jq '.Labels["com.redhat.component"]'
"vllm-rhel9-container"

# View image CVE scan results
$ oc get imagescan -n ai-inference
NAME                    CRITICAL   HIGH   MEDIUM   LOW   STATUS
vllm-rhel9-0.8.4-12    0          0      2        5     Passed

FIPS 140-3 Validation

Red Hat Enterprise Linux and OpenShift support FIPS 140-3 validated cryptographic modules. When OpenShift is installed in FIPS mode, all TLS connections (including those between inference pods, the EPP, and external clients) use FIPS-validated cryptographic implementations. This is a hard requirement for U.S. federal agencies and many financial institutions. The vLLM inference engine runs on top of RHEL's FIPS-validated OpenSSL, so inference API traffic is encrypted with validated cryptography without any application-level configuration.

CVE Response in Practice

When a critical vulnerability was discovered in vLLM's tokenizer parsing (CVE-2026-XXXX), the response timeline demonstrated the value of the enterprise security lifecycle:

By contrast, an organization running upstream vLLM from Docker Hub would need to: monitor the upstream repository for the security advisory, assess its applicability, rebuild the container image with the patch, push it to their registry, and coordinate a rollout across all clusters. This typically takes days to weeks depending on the organization's release processes.

Air-Gapped Deployment Security

For environments with no internet connectivity (defense, classified, certain financial), Red Hat AI supports fully air-gapped deployment. The oc-mirror tool creates an offline mirror of all required container images, operator bundles, and signatures. Model weights are pre-loaded onto encrypted portable storage and transferred to the PVC through a secure data diode process. All images in the air-gapped registry retain their original signatures, ensuring chain-of-custody verification even without network connectivity to registry.redhat.io.

Product Lifecycle Policy #

Enterprise procurement requires predictable support timelines. Red Hat AI follows Red Hat's standard product lifecycle policy, with defined support phases and end-of-life dates published at the time of each release's general availability.

Support Phases

• Full Support (18 months) -- Active development, bug fixes, security patches, and new feature backports. All severity levels accepted for support cases.
• Extended Update Support / EUS (36 months total) -- Security patches and critical bug fixes only. No new features. Available for designated EUS releases, allowing organizations to stay on a single version for up to 3 years.
• End of Life -- No further updates or support. Organizations must migrate to a supported version.

Version Timeline

Upgrade Paths

Red Hat AI supports sequential minor version upgrades (3.0 to 3.3, then 3.3 to 3.4). Skipping minor versions is not supported. The upgrade process uses OLM (Operator Lifecycle Manager) channel switching:

# Update the subscription channel
$ oc patch subscription redhat-ai-operator -n openshift-operators \
  --type=merge -p '{"spec":{"channel":"stable-3.4"}}'

# Monitor the upgrade
$ oc get csv -n openshift-operators -w
NAME                         DISPLAY      VERSION   PHASE
redhat-ai-operator.v3.3.2    Red Hat AI   3.3.2     Replacing
redhat-ai-operator.v3.4.0    Red Hat AI   3.4.0     Installing
redhat-ai-operator.v3.4.0    Red Hat AI   3.4.0     Succeeded

The operator upgrade does not restart inference pods. Existing LLMInferenceService resources continue running on the previous vLLM image until the operator reconciles them with the new version. This can be controlled with the spec.serving.image field: if pinned to a specific tag, the operator will not update it automatically. If set to a floating tag (not recommended for production), the operator will roll out the new image during reconciliation.

Day-2 Operations: Upgrades, Patching, Rollbacks, and Blue-Green Deployments #

Getting an inference service running is Day-1. Keeping it running, updating it safely, recovering from bad updates, and scaling it as demand grows -- that is Day-2, and it is where most operational pain lives. This section covers the Day-2 operations that enterprise teams perform regularly, with concrete procedures for each.

Rolling Upgrades

When a new vLLM image is published (for example, vllm-rhel9:0.8.5-1 with performance improvements or new model support), the upgrade process must happen without dropping a single inference request. The LLMInferenceService controller handles this through a rolling update strategy that integrates with PodDisruptionBudgets.

The process works as follows: the controller creates new pods with the updated image, waits for each new pod to pass its readiness probe (which includes model loading and a test inference), and only then terminates an old pod. The PDB ensures that the minimum number of replicas specified in pdb.minAvailable remains serving throughout the upgrade. For a deployment with 6 replicas and minAvailable: 4, the controller upgrades one pod at a time, maintaining at least 4 healthy replicas at every point.

# Update the image in the LLMInferenceService
$ oc patch llminferenceservice llama-3-70b-disagg -n ai-inference \
  --type=merge -p '{"spec":{"serving":{"image":"registry.redhat.io/redhat-ai/vllm-rhel9:0.8.5-1"}}}'

# Watch the rolling update progress
$ oc rollout status deployment/llama-3-70b-disagg-decode -n ai-inference
Waiting for deployment "llama-3-70b-disagg-decode" rollout to finish: 1 out of 4 new replicas have been updated...
Waiting for deployment "llama-3-70b-disagg-decode" rollout to finish: 2 out of 4 new replicas have been updated...
deployment "llama-3-70b-disagg-decode" successfully rolled out

# Verify all pods are running the new image
$ oc get pods -n ai-inference -l llm-d.ai/service=llama-3-70b-disagg \
  -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.containers[0].image}{"\n"}{end}'
llama-3-70b-disagg-prefill-0    registry.redhat.io/redhat-ai/vllm-rhel9:0.8.5-1
llama-3-70b-disagg-prefill-1    registry.redhat.io/redhat-ai/vllm-rhel9:0.8.5-1
llama-3-70b-disagg-decode-0     registry.redhat.io/redhat-ai/vllm-rhel9:0.8.5-1
llama-3-70b-disagg-decode-1     registry.redhat.io/redhat-ai/vllm-rhel9:0.8.5-1
llama-3-70b-disagg-decode-2     registry.redhat.io/redhat-ai/vllm-rhel9:0.8.5-1
llama-3-70b-disagg-decode-3     registry.redhat.io/redhat-ai/vllm-rhel9:0.8.5-1

Patching Strategy

There are two distinct types of patches in the Red Hat AI stack, and they follow different update paths:

Operator patches are delivered through OLM (Operator Lifecycle Manager). When Red Hat publishes a new operator version (for example, redhat-ai-operator.v3.4.1), OLM detects the update in the catalog, creates an InstallPlan, and replaces the operator pod. If your Subscription has installPlanApproval: Automatic, this happens without intervention. If set to Manual, an administrator must approve the InstallPlan before the upgrade proceeds. For production clusters, Manual approval is recommended so that patches are applied on your schedule, not on Red Hat's publication schedule.

Inference image patches are container image updates to the vLLM runtime. These are not managed by OLM -- they are controlled by the spec.serving.image field in your LLMInferenceService. When you pin to a specific tag (which you should always do in production), the image does not change until you explicitly update the tag. This separation means operator patches and inference patches can be applied independently, reducing change risk.

# Step 1: Apply the new image to staging first
$ oc patch llminferenceservice llama-3-70b-staging -n ai-staging \
  --type=merge -p '{"spec":{"serving":{"image":"registry.redhat.io/redhat-ai/vllm-rhel9:0.8.4-14"}}}'

# Step 2: Run a load test against staging
$ python benchmark_serving.py \
  --base-url https://$(oc get route llama-3-70b-staging -n ai-staging -o jsonpath='{.spec.host}') \
  --model meta-llama/Llama-3.1-70B-Instruct \
  --num-prompts 1000 \
  --concurrency 50

# Step 3: Compare latency metrics against baseline
$ oc exec -n openshift-monitoring prometheus-k8s-0 -- \
  promtool query instant http://localhost:9090 \
  'histogram_quantile(0.99, sum(rate(vllm:time_to_first_token_seconds_bucket{namespace="ai-staging"}[5m])) by (le))'

# Step 4: If metrics look good, apply to production
$ oc patch llminferenceservice llama-3-70b-disagg -n ai-inference \
  --type=merge -p '{"spec":{"serving":{"image":"registry.redhat.io/redhat-ai/vllm-rhel9:0.8.4-14"}}}'

Rollback Procedures

When an upgrade goes wrong -- a new vLLM image introduces a regression, a CRD schema change breaks a workflow, or an operator update causes unexpected behavior -- you need to roll back quickly. The rollback procedure depends on what went wrong.

Rolling back a vLLM image: Simply patch the LLMInferenceService to point back to the previous image tag. The controller will perform a rolling update back to the known-good version. This is the most common rollback scenario and the easiest to execute.

# Roll back to the previous known-good image
$ oc patch llminferenceservice llama-3-70b-disagg -n ai-inference \
  --type=merge -p '{"spec":{"serving":{"image":"registry.redhat.io/redhat-ai/vllm-rhel9:0.8.4-12"}}}'

# Monitor rollback progress
$ oc rollout status deployment/llama-3-70b-disagg-decode -n ai-inference

# Verify all pods are back on the old image
$ oc get pods -n ai-inference -l llm-d.ai/service=llama-3-70b-disagg \
  -o jsonpath='{range .items[*]}{.spec.containers[0].image}{"\n"}{end}' | sort -u
registry.redhat.io/redhat-ai/vllm-rhel9:0.8.4-12

Rolling back an operator version via OLM: This is more involved. OLM does not natively support downgrades. The recommended procedure is to delete the current Subscription, delete the ClusterServiceVersion (CSV), and create a new Subscription pinned to the previous version. Existing LLMInferenceService resources and their child resources (pods, services, etc.) are not affected by the operator deletion because the resources already exist -- they only become unmanaged until the previous operator version is reinstalled.

# Step 1: Delete the current subscription
$ oc delete subscription redhat-ai-operator -n openshift-operators

# Step 2: Delete the current CSV
$ oc delete csv redhat-ai-operator.v3.4.1 -n openshift-operators

# Step 3: Create a new subscription pinned to the previous version
$ oc apply -f - <<EOF
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: redhat-ai-operator
  namespace: openshift-operators
spec:
  channel: stable-3.4
  installPlanApproval: Manual
  name: redhat-ai-operator
  source: redhat-operators
  sourceNamespace: openshift-marketplace
  startingCSV: redhat-ai-operator.v3.4.0
EOF

# Step 4: Approve the install plan for the pinned version
$ oc get installplan -n openshift-operators
$ oc patch installplan install-xxxxx -n openshift-operators \
  --type=merge -p '{"spec":{"approved":true}}'

Blue-Green Deployments for Inference

For organizations that need zero-risk deployments, blue-green (or A/B) deployment patterns allow running two versions of a model side by side and gradually shifting traffic between them. This is particularly valuable when deploying a new LoRA adapter or a re-quantized model where you want to validate inference quality at production scale before committing.

OpenShift Routes support weighted traffic splitting natively. By creating two LLMInferenceService instances (blue and green) and a Route with weighted backends, you can control exactly what percentage of traffic goes to each version.

apiVersion: route.openshift.io/v1
kind: Route
metadata:
  name: llama-3-70b-production
  namespace: ai-inference
  annotations:
    haproxy.router.openshift.io/balance: roundrobin
spec:
  host: inference.example.com
  tls:
    termination: edge
  to:
    kind: Service
    name: llama-3-70b-blue                        # Current production (blue)
    weight: 90                                     # 90% of traffic
  alternateBackends:
    - kind: Service
      name: llama-3-70b-green                    # New version (green)
      weight: 10                                   # 10% canary traffic

# Start with 10% to green, monitor for 1 hour
$ oc set route-backends llama-3-70b-production -n ai-inference \
  llama-3-70b-blue=90 llama-3-70b-green=10

# If metrics look good, increase to 50%
$ oc set route-backends llama-3-70b-production -n ai-inference \
  llama-3-70b-blue=50 llama-3-70b-green=50

# Full cutover to green
$ oc set route-backends llama-3-70b-production -n ai-inference \
  llama-3-70b-blue=0 llama-3-70b-green=100

# If something goes wrong at any stage, instant rollback
$ oc set route-backends llama-3-70b-production -n ai-inference \
  llama-3-70b-blue=100 llama-3-70b-green=0

The blue-green pattern is especially powerful for A/B testing inference quality. You can compare output quality metrics (user satisfaction scores, task completion rates, hallucination rates) between the two versions at production scale, with real user traffic, before committing to the new version. The instant rollback capability -- a single command to shift 100% of traffic back to blue -- means the blast radius of a bad deployment is limited to the canary percentage.

Capacity Planning and Scaling

GPU capacity planning for inference is fundamentally different from CPU capacity planning. GPUs cannot be overcommitted the way CPUs can -- when GPU memory is full, requests queue or fail. There is no swap space. This makes proactive capacity planning essential.

The key monitoring signals that predict capacity exhaustion are:

• KV-cache utilization trend -- If average KV-cache utilization across replicas is trending upward over days or weeks, traffic is growing faster than capacity. When it sustains above 75% during peak hours, begin capacity expansion planning.
• Autoscaler ceiling hits -- If the HPA is consistently at maxReplicas during peak hours, you are capacity-constrained. The autoscaler cannot add more replicas even when demand warrants it. Either increase maxReplicas (if GPU capacity exists) or add GPU nodes.
• Queue depth during peaks -- A sustained queue depth greater than zero during expected peak periods indicates that current capacity is insufficient for your traffic pattern. Transient spikes are normal; sustained queues are a capacity signal.
• GPU node utilization -- Track nvidia.com/gpu allocation across the cluster. When total allocated GPUs exceed 80% of total available GPUs, order new hardware. GPU procurement lead times are typically 8-16 weeks, so plan accordingly.

The decision between adding nodes to an existing cluster versus adding a new cluster depends on your scale and failure domain requirements. For most organizations, adding nodes to an existing cluster is simpler -- the GPU Operator automatically configures new nodes, and the autoscaler can immediately schedule pods on them. Adding a new cluster makes sense when you need geographic distribution (a US-West cluster for West Coast latency), when a single cluster's control plane is hitting limits (typically around 5,000 nodes), or when regulatory requirements mandate separate failure domains.

# Check cluster-wide GPU allocation
$ oc get nodes -l nvidia.com/gpu.present=true \
  -o custom-columns=NAME:.metadata.name,GPUS:.status.capacity.nvidia\\.com/gpu,ALLOCATED:.status.allocatable.nvidia\\.com/gpu

# Check if autoscaler is hitting ceiling
$ oc get hpa -n ai-inference
NAME                     REFERENCE                       TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
llama-3-70b-disagg-hpa   Deployment/llama-3-70b-decode   85%/70%   2         12        12         30d

# Query capacity utilization trend over 7 days
$ curl -s "http://prometheus:9090/api/v1/query?query=avg_over_time(vllm:gpu_cache_usage_perc{namespace='ai-inference'}[7d])"

Enterprise vs. Upstream: A Detailed Comparison #

Choosing between upstream llm-d and Red Hat AI is not a binary decision. Most mature organizations use both, selecting the right tool for each stage of their workflow. The following table details the specific differences across every operational dimension:

"The TCO analysis showed that the Red Hat subscription paid for itself in avoided downtime costs within the first quarter."

-- VP of Infrastructure, Healthcare Tech

The sweet spot for many organizations is to use upstream llm-d in development and staging environments where engineers need access to the latest features and the freedom to experiment with different configurations. When a workload is ready for production -- when it needs to serve real users with contractual SLAs -- it moves to Red Hat AI, where the same CRD-based API ensures that the deployment manifest transfers directly from staging to production with minimal changes.

This is not about upstream being inferior. It is about recognizing that production infrastructure has requirements that extend far beyond feature functionality. When an inference endpoint is serving a customer-facing application and something breaks at 3 AM, the difference between "file a GitHub issue" and "call Red Hat support and get an engineer on the phone within an hour" can mean the difference between a minor incident and a P1 outage.

Self-Managed llm-d vs Red Hat AI Inference Server vs Cloud LLM APIs: A Complete Comparison #

The previous section compared upstream llm-d against Red Hat AI. But enterprise architects evaluating LLM inference options face a three-way decision: self-managed open-source llm-d on vanilla Kubernetes, Red Hat AI (the enterprise product), or cloud LLM APIs (OpenAI API, AWS Bedrock, Google Vertex AI). Each option occupies a different point on the cost-control-complexity spectrum. Here is how they compare across every dimension that matters for procurement.

Pricing Model Comparison

The economics of LLM inference change dramatically with scale. Cloud APIs charge per token, making them the cheapest option at low volumes and the most expensive at high volumes. Self-hosted options (both upstream llm-d and Red Hat AI) have high fixed costs (GPU hardware, power, cooling, engineering time) but near-zero marginal costs per token. The crossover point -- where self-hosted becomes cheaper than API -- depends on your utilization rate, model size, and token volume.

The Crossover Point

The critical insight from this comparison is that the cost curves cross. At low volumes (under 100M tokens/month), cloud APIs are dramatically cheaper because you avoid the fixed costs of GPU infrastructure entirely. But cloud API costs scale linearly with usage, while self-hosted costs scale sub-linearly (you can serve 10x more tokens by increasing batch sizes and utilization without 10x more GPUs).

At volumes above approximately 500M tokens per month, self-hosted inference beats cloud API pricing for equivalent model quality. Red Hat AI is the sweet spot for enterprises that need the cost savings of self-hosted infrastructure with the operational safety net of a vendor. The Red Hat subscription adds $50k-150k per year per cluster -- a fraction of the GPU hardware cost and an even smaller fraction of the cloud API cost at scale -- in exchange for 24/7 support, hardware certification, FIPS compliance, CVE response, and the operator automation that cuts operational burden in half.

Decision Matrix

Use this matrix to map your requirements to the right deployment approach. Green indicates full coverage, yellow indicates partial or manual coverage, and red indicates a gap.

A Note on Hidden Costs

The pricing table above captures direct infrastructure and subscription costs, but several hidden costs shift the comparison further toward self-hosted at scale:

• API rate limits and throttling. Cloud APIs impose rate limits that can throttle your application during peak demand. OpenAI's tier-1 rate limit of 500 requests per minute may be insufficient for a customer-facing application. Upgrading to higher tiers requires volume commitments and enterprise agreements that add procurement overhead.
• Data egress charges. Sending prompts and receiving responses over the internet incurs bandwidth costs. For applications with large context windows (32k+ tokens per request), data transfer costs add 5-15% on top of per-token pricing.
• Fine-tuning lock-in. If you fine-tune a model through a cloud API, the fine-tuned weights typically cannot be exported. You are paying for customization that you do not own. With self-hosted inference, your LoRA adapters and fine-tuned weights are assets on your infrastructure that you fully control.
• Outage exposure. When a cloud API experiences an outage, your application is down and you have no recourse beyond waiting. With self-hosted inference, your team controls the recovery timeline.

Decision Framework: Upstream vs. Enterprise #

The comparison table provides the raw data. This framework helps you apply it to your specific situation. Work through each question with your platform engineering, security, and procurement teams.

Migration Guide: Moving from Standalone vLLM to llm-d on Red Hat AI #

Many organizations start with standalone vLLM -- either running directly on bare metal, inside Docker, or deployed manually as Kubernetes Deployments. Migration to llm-d on Red Hat AI is straightforward because llm-d uses vLLM as its inference engine. Your model, your prompt templates, and your client integrations remain the same. What changes is how the deployment is managed, how routing works, and how the system scales.

"We migrated 12 standalone vLLM instances in a single sprint. Zero client-side changes."

-- Principal Engineer, SaaS Platform

What Changes

• Deployment mechanism -- Instead of managing Deployments, Services, ConfigMaps, and HPA resources manually, you define a single LLMInferenceService CRD. The controller creates all downstream resources.
• Routing -- Instead of a generic Kubernetes Service or Ingress with round-robin, requests flow through the EPP, which understands KV-cache state, LoRA affinity, and prefix caching.
• Autoscaling -- Instead of HPA on CPU utilization, autoscaling reacts to inference-specific metrics like queue depth and KV-cache utilization.
• GPU management -- Instead of manual driver installation, the GPU Operator manages drivers, device plugins, and topology discovery.
• Monitoring -- Instead of custom Prometheus configurations, metrics are scraped automatically through operator-created ServiceMonitors.

What Stays the Same

• API compatibility -- llm-d exposes the same OpenAI-compatible /v1/chat/completions and /v1/completions endpoints. Existing client code works without modification. Any SDK or library that works with the OpenAI API (including the official OpenAI Python and Node.js libraries, LangChain, LlamaIndex) works out of the box -- just change the base URL.
• Model format -- Same HuggingFace model weights. No conversion or re-quantization needed.
• vLLM engine -- The same vLLM engine powers inference. Sampling parameters, tokenizer behavior, and generation quality are identical.
• Prompt templates -- Chat templates, system prompts, and tool definitions work identically.

Config Mapping: vLLM CLI Arguments to CRD Fields

If you are currently launching vLLM with CLI arguments, here is how those map to LLMInferenceService CRD fields:

Migration Steps

Example: Before and After

Before -- launching vLLM directly:

$ python -m vllm.entrypoints.openai.api_server \
    --model meta-llama/Llama-3.1-70B-Instruct \
    --tensor-parallel-size 4 \
    --max-model-len 32768 \
    --dtype bfloat16 \
    --gpu-memory-utilization 0.9 \
    --host 0.0.0.0 \
    --port 8000

After -- the equivalent as an LLMInferenceService CRD, plus intelligent routing, autoscaling, monitoring, PDB, security hardening, and GPU lifecycle management:

apiVersion: serving.llm-d.ai/v1alpha1
kind: LLMInferenceService
metadata:
  name: llama-3-70b
  namespace: ai-inference
spec:
  model:
    name: meta-llama/Llama-3.1-70B-Instruct
    source:
      uri: pvc://model-store/llama-3-70b
    maxModelLen: 32768
    dtype: bfloat16
  serving:
    runtime: vllm
    image: registry.redhat.io/redhat-ai/vllm-rhel9:0.8.4-12
    serviceAccountName: inference-sa
    replicas: 1
    gpuMemoryUtilization: 0.9
    resources:
      gpu: nvidia.com/gpu
      gpuCount: 4
    securityContext:
      runAsNonRoot: true
      readOnlyRootFilesystem: true
      allowPrivilegeEscalation: false
  pdb:
    minAvailable: 1
  routing:
    endpoint: EPP
    strategy: kv-cache-aware
  autoscaling:
    minReplicas: 1
    maxReplicas: 8
    targetUtilization: 70

The CRD captures everything from the CLI command, and adds routing intelligence, autoscaling, PodDisruptionBudgets, security hardening, and integration with the OpenShift monitoring and GPU management stack. The total lines of configuration are roughly the same, but the operational burden shifts from you to the controller.

Case Study: How a Fortune 500 Bank Deployed llm-d on OpenShift AI #

To make this concrete, here is a composite case study drawn from patterns observed across multiple enterprise deployments. The specific numbers are realistic but the organization is fictional.

The Organization

A top-10 U.S. bank by assets, regulated by the Office of the Comptroller of the Currency (OCC) and the Federal Reserve. The bank's digital banking platform serves 12 million customers, and its wealth management division manages $400 billion in assets for high-net-worth clients.

The Challenge

The bank's AI team was tasked with deploying a customer-facing AI assistant for wealth management clients -- a conversational interface that could answer questions about portfolio performance, explain market movements, generate personalized investment summaries, and assist with common transactions. The requirements were non-negotiable:

• Data residency: No customer data could leave the bank's private data centers. Cloud LLM APIs were immediately disqualified because prompts containing customer portfolio data would traverse the public internet to a third-party provider.
• FIPS 140-3 compliance: All cryptographic operations (TLS termination, data at rest, API authentication) must use FIPS-validated modules. The OCC's Technology Risk Handbook mandates this for customer-facing systems.
• SOC 2 audit trail: Every inference request and response must be logged with full attribution (user identity, timestamp, model version, adapter version) for compliance audit and regulatory examination.
• 99.95% uptime SLA: The wealth management division committed to this SLA with their institutional clients. The AI assistant could not degrade this commitment.
• Sub-300ms P99 TTFT: Wealth management clients expect responsive interactions. The product team set 300ms as the ceiling for time-to-first-token at the 99th percentile.

Initial Approach: Self-Managed vLLM

The bank's platform engineering team initially attempted a self-managed deployment using upstream vLLM on a vanilla Kubernetes cluster. They provisioned 16 H100 GPUs across 2 bare-metal nodes, installed NVIDIA drivers manually, configured vLLM as a Kubernetes Deployment, and built a custom load balancer for request routing.

The results were discouraging. It took 14 weeks to reach a staging-ready state. The custom load balancer did not understand KV-cache state, leading to uneven GPU utilization and inconsistent latency. Driver upgrades required manual node draining and 4+ hours of maintenance per node. And when the deployment entered the bank's internal security audit, it failed on three critical findings: the container images were unsigned and pulled from Docker Hub, the TLS stack was not using FIPS-validated modules, and there was no documented CVE response process with contractual SLAs. The security team blocked production deployment.

Pivot to Red Hat AI

The team pivoted to Red Hat AI, deploying on the bank's existing OpenShift 4.16 clusters. The results were dramatically different:

• Week 1: Installed Red Hat AI operator and GPU Operator. Applied ClusterPolicy for automated driver management. Verified GPU discovery and node labeling.
• Week 2: Deployed Llama 3.1 70B as an LLMInferenceService with disaggregated serving (2 prefill, 6 decode replicas). Configured 4 LoRA adapters for the bank's use cases. Validated endpoint with synthetic traffic.
• Week 3: Security audit passed. FIPS compliance verified via RHEL's FIPS mode. Container image signatures verified. CVE response SLA documented through Red Hat's Premium support agreement. SOC 2 audit logging configured through OpenShift's audit subsystem.
• Week 4: Production deployment across 3 clusters with blue-green traffic shifting. Load tested at 2x projected peak volume. Monitoring dashboards operational with TTFT, ITL, and KV-cache alerts configured.

Production Architecture

The final production architecture spans three OpenShift clusters for geographic redundancy and disaster recovery:

• US-East (primary): 24 H100 SXM GPUs (3 nodes, 8 GPUs each). Runs 2 prefill replicas and 6 decode replicas with 4-way tensor parallelism.
• US-West (secondary): 16 H100 SXM GPUs (2 nodes). Runs 2 prefill replicas and 4 decode replicas. Serves West Coast clients with lower latency.
• DR site: 8 H100 SXM GPUs (1 node). Warm standby with model weights pre-loaded. Can scale to full capacity within 15 minutes of a primary site failure.

The base model is Llama 3.1 70B Instruct, served with four LoRA adapters fine-tuned for specific banking functions: wealth management (portfolio analysis, market commentary), mortgage advisory (rate comparison, document summary), commercial lending (credit analysis, term sheet generation), and fraud detection (transaction pattern analysis, alert triage). The LoRA-affinity routing strategy ensures adapter-hot replicas handle the majority of requests without adapter swaps.

Results

After six months of production operation:

• 99.97% uptime -- exceeding the 99.95% SLA, with zero Sev-1 incidents attributed to the inference platform.
• P99 TTFT of 280ms -- consistently under the 300ms ceiling, with P50 TTFT of 85ms.
• 40% GPU cost reduction compared to the initial self-managed vLLM attempt. The savings came from three sources: disaggregated serving (separate prefill and decode pools with different scaling characteristics reduced total GPU count by 25%), KV-cache-aware routing (better utilization of existing cache reduced redundant computation by 10%), and LoRA adapter sharing (serving 4 use cases from a single base model instead of 4 separate model deployments saved 60% of GPU memory).
• Passed OCC technology risk examination -- the bank's technology risk management team presented the deployment to OCC examiners, who found no material findings related to the AI infrastructure.

"We spent 14 weeks trying to build production-grade inference infrastructure ourselves and failed our security audit. With Red Hat AI, we were in production in 4 weeks and passed the audit on the first attempt. The subscription cost was less than what we were spending per month on the engineering team trying to make the DIY approach work."

-- Head of AI Platform Engineering, Top-10 U.S. Bank

Conclusion #

The relationship between llm-d and Red Hat AI represents the open-source-to-enterprise pipeline operating at its best. Innovation happens in the CNCF community, where researchers, engineers, and operators collaborate on the hardest problems in distributed inference: KV-cache-aware routing, disaggregated prefill/decode scheduling, LoRA-affinity placement, and prefix-cache optimization. Enterprise hardening happens at Red Hat, where that innovation is tested against specific hardware, certified for compliance requirements, backed by 24/7 support, and wrapped in a product lifecycle that guarantees patches and updates for years.

What makes this pipeline particularly powerful for LLM inference is that the CRD-based API is shared. The same LLMInferenceService YAML that works on upstream Kubernetes works on OpenShift. The same monitoring queries that surface TTFT and KV-cache utilization in a dev cluster work in a production cluster. The same troubleshooting playbook applies whether you are running community images or Red Hat-signed images. The abstraction boundary is clean and the migration path is concrete.

As LLM inference becomes critical infrastructure -- as fundamental to the enterprise as databases, message queues, and service meshes -- the infrastructure patterns described in this article will become the standard. Declarative CRDs for model deployment. GPU-topology-aware scheduling. Inference-specific autoscaling. Monitoring that speaks the language of tokens, not HTTP response codes. Disaggregated serving topologies that independently optimize prefill and decode. Support SLAs that match the criticality of the workload. Security lifecycle processes that respond to CVEs in hours, not weeks. These are not experimental features; they are the operational requirements of production LLM workloads.

The enterprise path to distributed LLM inference is not about choosing between open source and commercial software. It is about using both, at the right stage, with the right expectations. Start with llm-d upstream for speed, experimentation, and access to cutting-edge features. Graduate to Red Hat AI when your workload needs contractual SLAs, hardware certification, FIPS compliance, predictable lifecycle support, and the confidence that someone has tested the entire stack before it reaches your cluster. Contribute improvements back to the community. This is the same model that made Linux the foundation of the internet, Kubernetes the foundation of cloud computing, and is now making llm-d the foundation of enterprise LLM inference.

My Recommendation

After working extensively with both the upstream and enterprise stacks, here is my direct advice:

If you are a startup or research lab with a team that loves debugging CUDA errors at midnight, use upstream llm-d. You will move faster, have access to bleeding-edge features days after they merge, and your engineers will learn the system deeply. Accept that you are trading operational risk for velocity.

If you are an enterprise with revenue-generating AI workloads, use Red Hat AI. The subscription cost is noise compared to your GPU spend, and the operational benefits -- automated GPU lifecycle, FIPS compliance out of the box, 1-hour Sev-1 support, CVE patches in 48 hours -- are worth multiples of the subscription price in avoided incidents and engineering time. I have seen too many organizations spend 6 months building infrastructure that the Red Hat AI operator provides on day one.

If you are evaluating cloud APIs vs. self-hosted, start with cloud APIs for prototyping, then model your cost at projected 12-month volumes. If the math shows self-hosted winning by more than 30%, make the move. If the math is close, factor in data sovereignty, customization requirements, and compliance needs -- those non-cost factors usually tip the decision toward self-hosted for enterprises.

If you are already running standalone vLLM, migrate to llm-d. The migration is mechanical (same engine, same API, same model weights), the routing intelligence alone improves throughput by 15-30%, and you gain autoscaling, monitoring, and PDB protection that you would otherwise need to build yourself. Whether you run it on upstream Kubernetes or Red Hat OpenShift depends on the decision framework above.

What's Coming Next

The llm-d and Red Hat AI roadmap includes several capabilities that will further widen the gap between purpose-built inference platforms and ad-hoc deployments:

• Speculative decoding (GA): Currently in Tech Preview, speculative decoding uses a smaller draft model to predict multiple tokens ahead, which are then verified in a single forward pass of the large model. This can improve decode throughput by 2-3x for well-matched draft/target model pairs, with no quality degradation. Expect GA in Red Hat AI 3.5.
• Multi-modal inference: Support for vision-language models (Llava, Qwen-VL, InternVL) and audio models through the same LLMInferenceService CRD. The model field will accept multi-modal model configurations, and the EPP will understand modality-specific routing requirements (for example, routing image-heavy requests to replicas with more GPU memory headroom).
• Edge deployment with MicroShift: For inference at the edge -- retail stores, factory floors, autonomous vehicles -- Red Hat is working on a MicroShift profile for llm-d that runs a stripped-down inference stack on a single node with 1-2 GPUs. This extends the same CRD-based API to edge locations, enabling a consistent deployment model from cloud to edge.
• InferencePool and InferenceGateway GA: The fleet-level management CRDs described earlier in this article will reach general availability, enabling enterprise-scale multi-model, multi-team inference platforms with centralized routing, rate limiting, and authentication.
• Automated LoRA lifecycle: Integration with InstructLab's fine-tuning pipeline will enable a continuous delivery workflow where fine-tuned LoRA adapters are automatically tested, promoted, and deployed to production inference services without manual intervention.

The inference infrastructure space is evolving rapidly, but the architectural patterns are stabilizing. Declarative model deployment, intelligent request routing, disaggregated serving, and inference-specific observability are no longer experimental -- they are table stakes. The organizations that adopt these patterns now will have a structural advantage as LLM inference becomes the backbone of enterprise AI applications.