Multi-Tenant GPU Isolation: What We Learned After the Third Outage

01 The 2:47 AM Wake-Up Call#

I still remember the exact Slack message that woke me up. It was 2:47 AM, the PagerDuty alert had fired, and the on-call engineer had already posted a screenshot of the Grafana dashboard in #gpu-cluster-incidents: a single red line climbing past 2,400ms where Acme Corp's p95 TTFT was supposed to live comfortably below 200ms. The root cause? CodeGen Platform's nightly batch job had quietly consumed 78% of cluster GPU capacity, and our shared FIFO queue could not tell the difference between a $50M revenue pipeline and a speculative code completion experiment.

That was the incident that changed everything. We had five tenants on a shared 12-pod GPU cluster: Acme Corp (production API, p95 TTFT < 200ms), MedAI Labs (healthcare partner API, p95 TTFT < 400ms), FinServ Team (internal financial tools, p95 e2e < 1500ms), CodeGen Platform (batch code generation, p95 e2e < 30s), and Research Squad (experimental workloads, best-effort). We thought a shared FIFO queue with some basic rate limiting would be enough. We were wrong.

Here is what we did not understand at first: if you have operated multi-tenant CPU web services, you have a mental model for resource isolation that does not transfer to GPU inference. The differences are not cosmetic - they are structural, and they drove every design decision we eventually made with llm-d's isolation architecture.

CPU web services are fundamentally friendly to multi-tenancy. A typical API request arrives, does some computation for 5 to 50 milliseconds, reads from a database, and returns. The operating system provides virtual memory with hardware-enforced isolation between processes. Containers add a namespace boundary. Kubernetes enforces CPU and memory limits via cgroups. If one tenant's request consumes too much memory, the OOM killer terminates that process without affecting neighbors. The blast radius of any single request is small, and the recovery is fast.

GPU inference breaks every one of these assumptions. The night CodeGen Platform took down Acme Corp's API taught us that lesson viscerally. The first time we saw this dashboard, I think three of us on the call said the same thing simultaneously: "We need a completely different approach."

Long-running requests were the first thing that burned us. A single LLM inference call generating 2,000 output tokens at 40 tokens per second takes 50 seconds. CodeGen Platform's batch requests were processing 128K-context documents, each one running for over two minutes. During that entire time, each request held GPU memory and occupied a slot in the inference engine's continuous batching scheduler. This is not a 5ms HTTP roundtrip that frees resources almost instantly.

Massive KV cache memory was the second surprise. Every in-flight request maintains a key-value cache that grows linearly with sequence length. For our 70B-parameter model serving a 32K-token context, the KV cache for a single request consumed over 2 GB of GPU memory. An 80 GB GPU could hold fewer than 40 such requests simultaneously, and that was before accounting for model weights, activation memory, and other overhead. When CodeGen Platform submitted thousands of 128K-context requests, each one requiring 5+ GB of KV cache, those pods filled up fast.

Non-preemptable GPU memory made recovery impossible without killing requests. There is no virtual memory on GPUs. There is no swap partition. When a request allocates KV cache space, that memory is locked until the request completes or is explicitly cancelled. You cannot page it out to make room. You cannot overcommit. If the GPU is full, new requests queue, including Acme Corp's requests, which had no way to tell the GPU "I am more important."

The autoregressive decode loop meant we could not just kill long-running requests cheaply. LLM text generation is autoregressive: each output token depends on all previous tokens. If you kill a request mid-stream to free resources, you lose all the computation done so far. The request must restart from scratch. During our incident, we considered manually killing CodeGen Platform's requests, but each restart would waste the 30 to 60 seconds of prefill compute already consumed.

Batch interference via continuous batching was the root cause we kept missing. Modern inference engines like vLLM use continuous batching to maximize GPU utilization - they dynamically add and remove requests from the running batch. But CodeGen Platform flooding the system with high-throughput, long-sequence requests dominated the scheduler, consuming batch slots that Acme Corp's latency-sensitive requests needed. The scheduler sees requests, not tenants. It did not know that Acme Corp's $50M API should take priority over CodeGen Platform's speculative experiment.

02 GPU as a Shared Resource: Why CPU Intuitions Fail#

After the third outage, we did a deep-dive into why our CPU-era isolation strategies had failed so completely. The answer took us down to the hardware level, and it reshaped how we thought about the problem. I remember sitting in a conference room with our platform team, whiteboarding the GPU memory model, and having that sinking realization: none of our existing playbooks applied here.

GPU Memory Is Not CPU Memory#

CPU RAM benefits from decades of virtual memory engineering. Each process gets its own virtual address space. The MMU translates virtual addresses to physical addresses with hardware-enforced isolation. Pages can be swapped to disk. Memory can be overcommitted. The kernel mediates all access.

GPU memory (HBM on modern accelerators) has none of this. The CUDA driver allocates physical memory regions directly. There is no address space isolation between CUDA contexts running on the same GPU, beyond what the driver enforces in software. There is no swap. When 80 GB of HBM is allocated, it is gone. The allocator cannot reclaim it without the owning process releasing it.

For LLM inference, the KV cache is the dominant consumer of GPU memory after model weights. A single inference engine process loads the model weights once (consuming 35 to 70 GB for a 70B-parameter model in FP16/FP8), and the remaining memory is used for KV caches across all concurrent requests. During the CodeGen Platform incident, we watched our monitoring dashboards as KV cache utilization climbed from 40% to 94% in under two minutes. By the time Acme Corp's requests started queuing, there was literally no memory left to allocate them into.

Compute Scheduling on the GPU#

GPU compute is scheduled at the kernel (GPU kernel, not Linux kernel) level. CUDA streams provide a mechanism for concurrent execution, but the GPU's streaming multiprocessors (SMs) are shared. CodeGen Platform's batch of 64 requests would launch a large matrix multiplication that saturated most or all SMs, leaving Acme Corp's smaller, latency-sensitive operations waiting in the hardware queue. The GPU does not preempt running kernels mid-execution the way a CPU preempts threads.

In practice, LLM inference engines run one process per GPU that handles all requests through a single set of CUDA streams. There is no per-tenant process isolation at the GPU level. Acme Corp's requests and CodeGen Platform's requests are batched together in the same forward pass. This is by design - it is what makes GPU inference efficient - but it means isolation must happen before the request reaches the GPU.

Hardware Approaches and Their Limitations#

The outage taught us to evaluate every hardware-level isolation option available. None of them solved our problem.

NVIDIA Multi-Instance GPU (MIG) partitions a single GPU into up to seven isolated instances, each with dedicated SMs, memory, and L2 cache. MIG provides strong hardware isolation, but the partitions are coarse-grained (a 7-way split of an A100 gives each instance roughly 10 GB of memory and a fraction of compute). For our 70B model, which required the full GPU's memory just for weights, MIG partitions were far too small.

NVIDIA Multi-Process Service (MPS) allows multiple CUDA contexts to share a GPU with better concurrency than default time-slicing. However, MPS does not provide memory isolation - a misbehaving process can still exhaust GPU memory and affect neighbors. This was exactly the failure mode that had been waking us up at 2:47 AM.

Time-slicing (the default when multiple processes use a GPU) simply round-robins GPU access among processes. It provides no memory isolation and no priority scheduling. Context switches are expensive on GPUs, making this approach even less efficient for LLM inference workloads.

03 Priority Scheduling: The Algorithm That Saved Us#

The scheduling algorithm is the heart of multi-tenant isolation. Get it wrong and either Acme Corp's production API suffers or Research Squad's experiments starve completely. After three outages, we deployed llm-d's EPP with a combination of scheduling primitives that finally gave us both fairness and strict priority enforcement. I noticed during our rollout that the shift was almost immediate: the Grafana dashboards went from a chaotic tangle of latency spikes to clean, predictable curves within hours of turning on the new scheduler.

Why Our FIFO Queue Was Killing Acme Corp#

Our original scheduler was first-in-first-out: requests processed in arrival order. FIFO works fine when all requests are equal, but with five tenants running wildly different workloads, they were never equal.

With FIFO, every Acme Corp request queued behind thousands of CodeGen Platform batch requests. Each batch request took 5 to 60 seconds. Across 12 pods, the per-pod queue depth averaged 833 requests. Acme Corp's SLO required p95 TTFT under 200ms. The queue wait time alone exceeded 10 minutes. Our SLO was not "breached" - it was obliterated.

Weighted Fair Queuing#

Weighted Fair Queuing (WFQ) assigns each tenant a weight proportional to their allocated share of capacity. When we deployed WFQ, we gave Acme Corp a weight of 8, MedAI Labs a weight of 4, FinServ Team a weight of 2, and CodeGen Platform and Research Squad a weight of 1 each. Under contention, Acme Corp gets eight times the throughput of Research Squad. When the cluster is lightly loaded, all tenants get served immediately regardless of weight - WFQ only activates when there is contention.

The EPP implements WFQ at the routing layer. Each tenant's weight maps to their priority level and configured capacity share. When selecting which queued request to route next, the EPP uses a virtual-time-based algorithm that ensures each tenant's actual throughput converges to their weighted share over time. This prevents starvation: even Research Squad, at the lowest weight, makes progress - just at a proportionally lower rate.

Priority Preemption: The Recovery#

WFQ handles steady-state contention, but sometimes you need hard priority: Acme Corp's production request arrives and every pod is busy with CodeGen Platform batch work. Waiting for a batch request to complete naturally might take 30 to 60 seconds. Acme Corp's SLO requires p95 TTFT under 200ms and p95 end-to-end under 500ms. This is where priority preemption saved us.

When a priority-5 (critical) request from Acme Corp arrives and the EPP determines that no pod can serve it within the 200ms TTFT window, it signals a pod to preempt its lowest-priority in-flight request. The preemption is graceful: the preempted request's partial output is discarded, and the request is re-queued at its original priority level. CodeGen Platform experiences a retry delay, not a failure.

Token Bucket Rate Limiting#

Rate limiting prevents any single tenant from flooding the system, regardless of their priority. We use the token bucket algorithm - a well-understood rate limiting primitive from networking - and if we had configured it properly before the CodeGen Platform incident, the incident would never have happened.

Each tenant has a bucket with a defined capacity (the burst allowance) and a fill rate (the sustained rate). Each request consumes one token from the bucket. When the bucket is empty, requests are rejected with a 429 status code until tokens replenish. CodeGen Platform's bucket has a burst size of 10 and a sustained rate of 100 req/s - enough for their batch processing needs, but physically incapable of the 10,000-request burst that caused the original outage.

Priority Queue Visualization#

The EPP maintains separate queues per priority level. Requests are dequeued in strict priority order: all Acme Corp P5 requests are served before any MedAI Labs P4 request, all P4 before FinServ Team's P3, and so on. Within the same priority level, requests follow WFQ ordering based on tenant weights.

Configuration: How We Set Up Our Five Tenants#

apiVersion: inference.networking.x-k8s.io/v1alpha2
kind: InferencePool
metadata:
  name: llama-70b-shared
spec:
  targetPortNumber: 8000
  selector:
    app: llama-70b

  tenants:
    - name: "acme-corp"
      priority: 5              # Critical: preempts all others
      weight: 8               # WFQ weight for fair share
      rateLimit:
        requestsPerSecond: 200
        burstSize: 50
      maxConcurrency: 100
      podAffinity: "dedicated"  # 4 dedicated pods
      slo:
        ttft_p95_ms: 200
        e2e_p95_ms: 500
        availability: 99.9

    - name: "medai-labs"
      priority: 4              # High: yields only to P5
      weight: 4
      rateLimit:
        requestsPerSecond: 50
        burstSize: 20
      maxConcurrency: 30
      slo:
        ttft_p95_ms: 400
        e2e_p95_ms: 800
        availability: 99.5

    - name: "finserv-team"
      priority: 3              # Normal: standard service level
      weight: 2
      rateLimit:
        requestsPerSecond: 30
        burstSize: 10
      maxConcurrency: 20
      slo:
        e2e_p95_ms: 1500
        availability: 99.0

    - name: "research-squad"
      priority: 2              # Low: best-effort access
      weight: 1
      rateLimit:
        requestsPerSecond: 10
        burstSize: 5
      maxConcurrency: 10
      slo:              # Best effort - no hard SLO
        enforcement: "none"

    - name: "codegen-platform"
      priority: 1              # Batch: fills unused capacity
      weight: 1
      rateLimit:
        requestsPerSecond: 100
        burstSize: 10          # Minimal burst - the 10K spike that caused the outage is now impossible
      maxConcurrency: 40
      slo:
        e2e_p95_ms: 30000      # 30s - batch tolerant
        availability: 95.0

04 Deep Dive: Fairness Algorithms and Their Trade-Offs#

After we had the basic priority scheduling and WFQ in place, I spent two straight weeks going deeper into the fairness algorithm space. Our team needed to understand not just what WFQ does, but why it works, where it breaks, and what alternatives we should consider. This section is the result of that investigation, and I wish I had written it before we chose our initial configuration instead of after.

Weighted Fair Queuing: The Virtual Time Algorithm#

WFQ is built on the concept of "virtual time" - a clock that advances differently for each tenant based on their weight. The core idea is elegant: imagine a hypothetical bit-by-bit round-robin system where each tenant sends data one bit at a time, weighted by their allocation. WFQ simulates this ideal system by computing when each request would have finished under that ideal model, then serving requests in order of their virtual finish times.

The virtual time calculation works as follows. The system maintains a global virtual time V(t) that represents the progress of the ideal fair scheduler. Each tenant i has a per-tenant virtual time S_i (the virtual start time of the next request) and a weight w_i. When a new request arrives from tenant i, the scheduler computes:

class WFQScheduler:
    def __init__(self):
        self.global_virtual_time = 0.0
        self.tenant_virtual_time = {}  # per-tenant virtual clock
        self.active_tenants = set()
        self.queue = PriorityQueue()  # ordered by virtual_finish_time

    def on_request_arrival(self, request, tenant_id, weight):
        # Virtual start time: max of tenant's own clock and global time
        # This prevents a tenant that was idle from "banking" credits
        S_i = max(
            self.tenant_virtual_time.get(tenant_id, 0.0),
            self.global_virtual_time
        )

        # Virtual finish time: start + (cost / weight)
        # Lower weight = higher finish time = served later
        cost = request.estimated_gpu_seconds
        F_i = S_i + (cost / weight)

        # Update tenant's virtual clock
        self.tenant_virtual_time[tenant_id] = F_i
        self.active_tenants.add(tenant_id)

        # Enqueue with virtual finish time as sort key
        self.queue.push(request, priority=F_i)

    def select_next_request(self):
        # Serve the request with the LOWEST virtual finish time
        request = self.queue.pop_min()

        # Advance global virtual time
        # Rate = 1 / sum(weights of active tenants)
        total_active_weight = sum(
            w for t, w in self.weights.items()
            if t in self.active_tenants
        )
        self.global_virtual_time += (
            request.actual_gpu_seconds / total_active_weight
        )

        return request

The critical subtlety is in the max() call on the virtual start time. When a tenant goes idle (say, Research Squad runs no experiments for an hour) and then returns, their virtual start time is set to the current global virtual time, not left at the stale value from when they were last active. Without this, a returning tenant would have a very low virtual finish time and effectively "jump the queue," getting a burst of service that starves other tenants. We discovered this behavior during testing when Research Squad would submit nothing for 45 minutes, then dump 200 requests at once and briefly starve FinServ Team's P3 queue. Setting the virtual start time to max(tenant_clock, global_clock) eliminates this problem entirely.

In our deployment, Acme Corp's weight of 8 means their requests accumulate virtual time at 1/8th the rate of CodeGen Platform's weight-1 requests. If both submit a request with the same estimated GPU cost, Acme Corp's virtual finish time will be 8x closer to the current virtual time, so it gets served first. Under contention with all five tenants active, Acme Corp gets 8/(8+4+2+1+1) = 50% of throughput, MedAI Labs gets 25%, FinServ Team gets 12.5%, and CodeGen Platform and Research Squad each get 6.25%.

Token Bucket Rate Limiting: The Burst Allowance Math#

The token bucket algorithm is deceptively simple but the math around burst allowance is worth understanding deeply, because getting it wrong is exactly what allowed CodeGen Platform to crash our cluster.

A token bucket has two parameters: the fill rate r (tokens per second) and the bucket capacity b (maximum tokens the bucket can hold). Tokens accumulate at rate r up to the maximum b. Each request consumes one token. The maximum burst duration - how long a tenant can sustain a burst above the fill rate - is calculated as:

burst_duration = b / (burst_rate - r)

For CodeGen Platform with b = 10 and r = 100/s: if they try to burst at 200 req/s, the burst lasts 10 / (200 - 100) = 0.1 seconds before they start getting 429s. If they try to burst at 110 req/s, the burst lasts 10 / (110 - 100) = 1 second. The bucket capacity is the control knob for how much above-rate traffic you tolerate.

Before the outage, CodeGen Platform had no rate limit at all. Their batch job client submitted requests as fast as the network would carry them, which peaked at roughly 3,000 req/s. With a bucket capacity of 10, their maximum burst above the 100/s sustained rate is physically limited to 10 extra requests. The 10,000-request spike scenario becomes structurally impossible.

Nested rate limits add another layer that we learned to appreciate. We run both per-tenant rate limits and a global cluster rate limit. The global limit is set at 400 req/s across all tenants combined. This prevents a scenario where all five tenants simultaneously burst to their individual limits (200 + 50 + 30 + 10 + 100 = 390 req/s sustained, but with bursts totaling 50 + 20 + 10 + 5 + 10 = 95 extra requests). The global bucket catches combined bursts that per-tenant limits would allow individually. In practice, we set the global bucket capacity to 60, roughly 2/3 of the sum of per-tenant burst sizes, because we observed that simultaneous bursts from more than three tenants were rare.

Priority Preemption: Cost Analysis#

Preemption is powerful but expensive. I have learned to think of every preemption event as a direct cost in wasted GPU-seconds. When a CodeGen Platform request is preempted after running for t seconds, you lose exactly t GPU-seconds of compute. For their 128K-context batch requests, the average preemption happens at the 45-second mark (roughly 60% through a typical 75-second request), wasting 45 GPU-seconds per event. At $0.007 per GPU-second, each preemption costs about $0.32 in wasted compute.

That sounds small until you consider preemption cascades. A preemption cascade occurs when preempting one request causes a chain reaction. Here is how it happens: the EPP preempts CodeGen Platform request C1 on pod 7 to make room for Acme Corp request A1. C1 is re-queued and assigned to pod 9. But pod 9 is also full, so the EPP preempts Research Squad request R1 on pod 9. R1 is re-queued to pod 3. If pod 3 is also full, another preemption occurs. We observed cascades of up to 4 deep during peak contention, wasting a total of 180 GPU-seconds (4 requests at an average of 45 seconds each) to serve a single Acme Corp request.

We tuned preemption thresholds to prevent cascades. The key parameter is the preemption depth limit, which we set to 2: the EPP will preempt at most 2 requests in a chain. If a 2-deep preemption chain does not free sufficient resources, the Acme Corp request is queued normally rather than triggering further cascades. We also added a preemption cooldown of 5 seconds per pod: after a pod has a request preempted, it is immune from further preemptions for 5 seconds. This prevents the same pod from being repeatedly targeted. After tuning these thresholds, our average wasted GPU-seconds per preemption event dropped from 90 to 52, and cascade events dropped from 12/day to fewer than 2/day.

Deficit Round Robin: The Alternative We Considered#

Before settling on WFQ, we seriously evaluated Deficit Round Robin (DRR) as an alternative. DRR is simpler to implement and has O(1) per-request overhead compared to WFQ's O(log n) priority queue operations. In DRR, each tenant gets a "quantum" proportional to their weight and a "deficit counter" that tracks unused allocation.

Each scheduling round, the scheduler visits each tenant in turn, adds their quantum to their deficit counter, and serves requests until the deficit counter is exhausted or the tenant's queue is empty. Unused deficit carries forward to the next round (up to a maximum to prevent unbounded accumulation).

DRR's advantage is predictable CPU overhead in the scheduler itself. Its disadvantage, and the reason we chose WFQ instead, is that DRR has worse latency behavior under bursty workloads. In DRR, a newly arrived high-weight request must wait for the current round to complete before it can be served. With 5 tenants and round times of 10 to 50ms depending on load, this adds up to 50ms of scheduling delay. For Acme Corp's 200ms TTFT target, burning 50ms in scheduler delay before the request even reaches a pod was unacceptable. WFQ, by contrast, immediately computes the virtual finish time and can serve the request next if its virtual finish time is the lowest. The trade-off is higher scheduler CPU cost, but at our scale of 400 req/s, the scheduler CPU is not the bottleneck - GPU is.

05 SLO Enforcement: From Aspirational to Contractual#

Priority scheduling determines who goes first. SLO enforcement determines whether "first" is actually fast enough. Before the outages, our SLOs were aspirational - "Acme Corp should be fast." After, they became contracts with specific numbers, real-time tracking, and automated enforcement. The difference between those two things is the difference between hoping and knowing. I remember the meeting where we switched from aspirational SLOs to contractual ones. Our engineering director asked, "What number would make you comfortable going back to sleep when paged?" That question cut through months of vague discussions.

The SLO Contracts We Run Today#

A meaningful SLO specifies a metric, a percentile, and a threshold. "Average latency should be low" is not an SLO. Here is what we actually commit to for each tenant:

The distinction between "proactive" and "reactive" enforcement is something we learned to care about deeply. Averages hide tail latency. A service with 50ms average latency might have a p99 of 5 seconds if a small percentage of requests get stuck behind long-running batch work. During the CodeGen Platform incident, Acme Corp's average latency was only 380ms - but the p95 had spiked to 2,400ms. If we had been watching the average, we would not have known there was a problem until customers started calling.

Real-Time Latency Tracking#

The EPP tracks per-tenant latency percentiles in real-time using a sliding window of the last 60 seconds. For each tenant, it maintains:

• p50, p95, and p99 TTFT - Acme Corp's p95 TTFT typically runs at 140ms, giving us 60ms of headroom before the 200ms threshold
• p50, p95, and p99 end-to-end latency - MedAI Labs' p95 e2e typically runs at 620ms against an 800ms threshold
• Throughput (requests completed per second) - Acme Corp averages 145 req/s during peak hours
• Queue depth per priority level - CodeGen Platform's P1 queue can safely grow to 200+ without impacting higher tiers
• Active concurrency (in-flight requests) per tenant - alerts fire when Acme Corp exceeds 80 concurrent requests

These metrics feed directly into routing decisions. When the EPP needs to assign a request to a pod, it considers not just the pod's current load, but whether routing to that pod would push the tenant's latency percentiles past their SLO threshold.

Proactive vs. Reactive: The Insight That Changed Our Architecture#

Here is what we did not understand until the second outage: a reactive system detects an SLO breach after it happens and then scrambles to recover. By the time you see "Acme Corp p95 TTFT = 350ms" in your dashboard, you have already breached, and dozens of requests have been slow. A proactive system detects that a breach is imminent and adjusts routing before it occurs.

Concretely: when Acme Corp's p95 TTFT is currently 160ms (against a 200ms threshold, with 20% headroom triggering at 160ms), and the EPP observes that CodeGen Platform's queue depth is rising rapidly, it starts steering Acme Corp requests to less-loaded pods, even if those pods are not the optimal choice for KV cache reuse. The SLO takes precedence over cache efficiency. If pressure continues, the EPP begins borrowing capacity from CodeGen Platform and Research Squad by throttling their request rates. If the situation still does not resolve, preemption activates.

SLO Configuration#

tenants:
  - name: "acme-corp"
    slo:
      timeToFirstToken:
        p95Ms: 200            # The number that woke us up at 2:47 AM
        p99Ms: 500            # Hard limit - page if breached
      endToEndLatency:
        p95Ms: 500
        p99Ms: 1200
      availability: 99.9
      throughput:
        minRequestsPerSecond: 100  # Guaranteed minimum
      enforcement:
        mode: "proactive"     # Adjust before breach, not after
        headroomPercent: 20  # Start adjusting at 160ms (80% of 200ms)

  - name: "medai-labs"
    slo:
      timeToFirstToken:
        p95Ms: 400
        p99Ms: 800
      endToEndLatency:
        p95Ms: 800
        p99Ms: 1500
      availability: 99.5
      enforcement:
        mode: "proactive"

  - name: "finserv-team"
    slo:
      endToEndLatency:
        p95Ms: 1500
      availability: 99.0
      enforcement:
        mode: "reactive"

  - name: "codegen-platform"
    slo:
      endToEndLatency:
        p95Ms: 30000          # 30 seconds - batch is tolerant
      availability: 95.0
      enforcement:
        mode: "best-effort"   # No preemption or capacity borrowing

  - name: "research-squad"
    slo:
      enforcement:
        mode: "none"          # Best effort - no hard SLO

Prometheus Metrics: The Dashboard We Wish We Had from Day One#

If we had this instrumentation during the first outage, we would have caught the problem 45 minutes before it woke anyone up. The EPP exposes per-tenant metrics through a standard Prometheus endpoint:

# Per-tenant latency histogram
llmd_epp_request_duration_seconds{tenant="acme-corp",phase="ttft"}
llmd_epp_request_duration_seconds{tenant="medai-labs",phase="e2e"}

# Per-tenant active concurrency
llmd_epp_active_requests{tenant="acme-corp"}
llmd_epp_active_requests{tenant="codegen-platform"}

# Per-tenant rate limit state
llmd_epp_rate_limit_tokens_remaining{tenant="codegen-platform"}
llmd_epp_rate_limit_rejections_total{tenant="codegen-platform"}

# Per-tenant SLO compliance - the number we watch most closely
llmd_epp_slo_compliance_ratio{tenant="acme-corp",metric="ttft_p95"}  # Target: > 0.999
llmd_epp_slo_compliance_ratio{tenant="medai-labs",metric="ttft_p95"}  # Target: > 0.995

# Preemption events - non-zero means cluster is saturated
llmd_epp_preemptions_total{preempted_tenant="codegen-platform",preempting_tenant="acme-corp"}

# Cost attribution counter
llmd_epp_gpu_seconds_total{tenant="acme-corp"}
llmd_epp_gpu_seconds_total{tenant="codegen-platform"}

06 Cost Attribution: What the CFO Wants to See#

Before we implemented cost attribution, our GPU infrastructure was a $54,000/month black box line item. Every team wanted the best GPUs, the highest priority, and the most capacity - because someone else was paying for it. The CodeGen Platform team had no idea their nightly batch job consumed 47% of total GPU-hours. The Research Squad team assumed their experiments were "basically free." Nobody had any incentive to use resources efficiently because nobody could see what they were using. The first time we showed a team lead the actual numbers, the room went quiet for about ten seconds.

Cost attribution closed this loop. It made each team's consumption visible and chargeable, and it fundamentally changed how every team lead thought about GPU time.

Why GPU-Seconds#

The natural unit for GPU cost attribution is the GPU-second: the amount of time a tenant's requests occupied GPU compute and memory resources. One GPU-second on an H100 costs roughly $0.007 - which seems small until you realize that CodeGen Platform consumed 44,640,000 GPU-seconds in a single month. At that scale, even small efficiency improvements translate to meaningful cost savings.

GPU-seconds capture both compute and memory occupancy. A request that is actively decoding tokens consumes GPU compute. A request that is waiting in the continuous batching scheduler with its KV cache loaded consumes GPU memory. Both are real resource consumption, and both are tracked.

How llm-d Tracks Consumption#

The EPP logs the start time, end time, pod assignment, and tenant identity for every request it routes. From these logs, you can compute:

• Wallclock GPU-seconds: (request_end_time - request_start_time) for each request, summed per tenant - Acme Corp averaged 29,520,000 GPU-seconds/month
• Weighted GPU-seconds: wallclock time weighted by the fraction of GPU resources the request consumed (approximated by sequence length relative to maximum batch capacity)
• Pod-hours: the total time a tenant had requests occupying a specific pod - critical for Acme Corp's dedicated pod chargeback

The Monthly Cost Report That Changed Everything#

Before this report existed, GPU infrastructure was a black box line item on the platform team's budget. After, every team lead could see exactly what they were spending and why. The first time we circulated this report, three things happened: CodeGen Platform's team lead immediately asked about off-peak scheduling to reduce their rate, Research Squad voluntarily reduced their experiment frequency, and Acme Corp stopped arguing about the infrastructure surcharge on their P5 priority tier because they could see the dedicated pod allocation they were paying for.

Notice that CodeGen Platform consumes the most GPU-hours (12,400) but pays the least per hour ($1.00). Their work runs at the lowest priority, filling capacity that would otherwise sit idle. This pricing structure incentivizes teams to honestly classify their workloads: if CodeGen Platform requested P5 priority for a job that can tolerate P1 latency, they would pay $3.50/hr instead of $1.00/hr - $43,400/month instead of $12,400/month - for no meaningful performance benefit.

We ran shadow billing for the first month before turning on real chargeback. Every team lead received a "what you would have paid" report alongside the cluster-wide cost breakdown. This gave teams time to optimize their usage patterns before the costs became real. CodeGen Platform moved their heaviest batch jobs to the 10 PM to 6 AM window (when Acme Corp's traffic drops 90%) and reduced their effective cost by 22% without changing their priority tier.

07 Chargeback Models: Billing Internal Teams for GPU Usage#

Cost attribution tells you who used what. Chargeback determines who pays for what and how. These are fundamentally different problems, and getting chargeback wrong can be worse than having no chargeback at all. I learned this lesson when our first chargeback model, a flat per-request fee, caused CodeGen Platform to optimize for fewer, larger requests (cramming multiple documents into single 128K-context calls) which actually made our noisy neighbor problem worse, not better.

Tiered Pricing: Higher Priority Equals Higher Rate#

Our production chargeback model uses tiered pricing where the rate per GPU-hour increases with priority level. This is not arbitrary markup. Higher priority tiers consume more real resources because they come with guarantees: dedicated pod capacity, preemption rights, proactive SLO enforcement, and the operational overhead of monitoring and alerting infrastructure. The pricing reflects these costs.

Billing Granularity: Per-Token vs. Per-Request vs. Per-GPU-Second#

We evaluated three billing granularities before settling on GPU-seconds as our primary metric. Each has trade-offs that matter depending on your tenants' workload profiles.

Per-token billing charges based on the number of input and output tokens processed. This is the model most cloud LLM APIs use (OpenAI, Anthropic, etc.) and it has the advantage of being intuitive to developers. But it poorly reflects actual GPU cost: a 100-token request on a congested GPU takes far more wall-clock time (and thus more GPU-seconds) than the same request on an idle GPU. Per-token billing does not capture the contention cost, so tenants have no incentive to avoid peak hours.

Per-request billing charges a flat fee per API call regardless of token count. This is the simplest model to implement and explain, but it creates perverse incentives. CodeGen Platform started cramming multiple documents into single requests to reduce their per-request charges, which resulted in longer-running requests that were harder to preempt and worse for cluster health. We abandoned per-request billing after two weeks.

Per-GPU-second billing charges based on actual GPU time consumed. This aligns costs with real resource usage. A 50-second request costs 50x more than a 1-second request. A request that runs during congestion and takes longer costs more than one during off-peak. This naturally incentivizes tenants to optimize request efficiency and shift workloads to off-peak hours. It is the model we run in production.

Handling Shared Overhead#

Not all GPU time is attributable to a specific tenant. The shared overhead includes model loading (the initial model weight load takes 45 to 90 seconds per pod restart and consumes GPU memory during the entire pod lifecycle), warmup inference (the first 10 to 20 requests after model load run slower due to CUDA kernel compilation and cache warmup), and idle capacity (time when a GPU is powered on but no tenant has active requests).

We allocate shared overhead proportionally to each tenant's usage. If Acme Corp consumed 31.4% of total billable GPU-seconds in a month, they are charged 31.4% of the shared overhead pool. The overhead pool for our cluster is approximately $2,800/month (model reloads, idle time during the 3 AM to 6 AM lull, and warmup cycles). This adds roughly $880 to Acme Corp's monthly bill and $35 to Research Squad's, which feels proportionally fair.

Shadow Billing vs. Real Chargeback Rollout#

We strongly recommend running shadow billing for at least one full billing cycle before turning on real chargeback. During shadow billing, every team receives a detailed invoice showing exactly what they would have been charged, but no actual money changes hands. This gives teams time to react and optimize.

Our shadow billing month revealed several surprises. CodeGen Platform discovered that 23% of their GPU-hours were consumed by retry storms caused by aggressive timeout settings in their batch client. Fixing their timeout configuration reduced their projected bill by $2,852/month. Research Squad found that their hyperparameter sweep experiments were accidentally running on the production cluster instead of their development cluster - a routing misconfiguration that accounted for 40% of their usage. FinServ Team realized their morning report generation could be batched more efficiently by combining similar queries, reducing their request count by 35% and their projected cost by $540/month.

After the shadow month, we rolled out real chargeback with a 30-day grace period where overages triggered warnings but not actual charges. This phased approach converted every team into an active participant in resource optimization rather than a passive consumer.

Detailed Monthly Invoice Breakdown#

Here is what an actual monthly invoice looks like for CodeGen Platform, our highest-volume tenant. We send this level of detail to every team lead monthly.

Notice the off-peak discount. We offer a 30% rate reduction for GPU-hours consumed between 10 PM and 6 AM, when Acme Corp's traffic is at its lowest. This directly incentivized CodeGen Platform to shift their heaviest batch processing to overnight hours, which reduced contention during peak hours and improved Acme Corp's p95 TTFT by 12ms on average. The discount costs us $2,232/month in reduced revenue but saves us far more in avoided preemption waste and SLO breach risk.

How Chargeback Changed Team Behavior#

The behavioral changes from chargeback were more dramatic than I expected. Within two months of turning on real billing:

• CodeGen Platform shifted 60% of their batch processing to the 10 PM to 6 AM window, reducing their monthly bill from a projected $12,400 to $8,710. Their tech lead told me, "We never would have bothered optimizing if we were not paying for it."
• Research Squad implemented an experiment scheduler that automatically pauses non-critical runs when the cluster is above 70% utilization. Their monthly bill dropped from $900 to $540, and they stopped accidentally degrading FinServ Team's SLO during business hours.
• FinServ Team consolidated three separate report generation jobs into a single batch pipeline, reducing their request count by 35% and their cost by $540/month. They also started caching inference results for repeated queries.
• Acme Corp optimized their prompt templates to reduce average input length by 18%, which reduced per-request GPU time and saved roughly $1,200/month. Before chargeback, nobody on their team even knew what their average prompt length was.

Integration with FinOps Tools#

Our chargeback data integrates with Kubecost and OpenCost through the EPP's Prometheus metrics endpoint. The key metrics for FinOps integration are:

• llmd_epp_gpu_seconds_total{tenant="..."} - cumulative GPU-seconds per tenant, the foundation for all billing calculations
• llmd_epp_request_cost_dollars{tenant="...", priority="..."} - pre-calculated cost per request based on priority tier and time-of-day rates
• llmd_epp_preemption_waste_seconds{tenant="..."} - GPU-seconds wasted on preempted requests, critical for understanding the true cost of priority enforcement

Kubecost scrapes these metrics and displays them alongside standard Kubernetes resource costs (CPU, memory, storage) in its cost allocation dashboard. This gives finance teams a single pane of glass for all infrastructure costs, with GPU inference costs broken out per tenant. OpenCost provides similar functionality as a CNCF Sandbox project, which aligned well with our commitment to open-source tooling.

08 Security Isolation: The Question Every CISO Asks#

When we onboarded MedAI Labs - a healthcare partner handling HIPAA-regulated patient data - their CISO asked the question we knew was coming: "Can Acme Corp's prompts leak to MedAI Labs' responses, or vice versa?" We had done the engineering analysis, but until you can answer this question precisely and with evidence, you will not clear compliance review. I remember rehearsing our answer three times before the meeting because we knew one vague response would kill the deal. Here is the answer we gave, and the architecture that backs it up.

Inference Context Isolation#

LLM inference serving is stateless per-request at the application level. Each request gets its own KV cache allocation, its own attention computation, and its own output generation. There is no shared mutable state between Acme Corp's requests and MedAI Labs' requests. The model weights are read-only - they are loaded once and shared across all requests, but no request can modify them.

This is fundamentally different from training, where data can leak through gradient updates to model weights (the "memorization" problem). During inference, the model weights do not change. Acme Corp's prompt cannot influence the weights, and therefore cannot influence MedAI Labs' output. Each request's KV cache is allocated in a separate memory region and is deallocated when the request completes.

What About Continuous Batching?#

Continuous batching processes multiple requests in the same GPU kernel invocation for efficiency. Does this create a data leakage path? No. The attention computation is per-request: each request attends only to its own KV cache. Requests are concatenated along the batch dimension for compute efficiency, but the attention mask ensures that no cross-request attention occurs. This is a fundamental property of how transformer inference works, not something that needs to be bolted on. We provided MedAI Labs' security team with a detailed walkthrough of the vLLM attention kernel code to verify this independently.

Network-Level Isolation#

While the inference engine provides application-level isolation, defense-in-depth adds network-level controls. We deployed all of these after the MedAI Labs onboarding review:

• Kubernetes NetworkPolicies restrict which namespaces and pods can reach inference endpoints - MedAI Labs' namespace can only reach pods in their affinity group
• mTLS between the EPP and inference pods ensures request routing is authenticated and encrypted in transit
• Tenant identity propagation via JWT tokens ensures the EPP can verify tenant identity at every hop - Acme Corp's requests carry a signed x-tenant-id: acme-corp header
• Audit logging records which tenant accessed which endpoint, when, and from where - providing the forensic trail that MedAI Labs' compliance team required for HIPAA documentation

Compliance Considerations#

For organizations operating under SOC 2, HIPAA, or similar frameworks, the key documentation requirement is demonstrating that tenant data isolation is maintained throughout the request lifecycle. The combination of per-request KV cache isolation (application level), NetworkPolicies (network level), and audit logging (operational level) provides the evidence chain that auditors require. MedAI Labs additionally required dedicated pod pools for their most sensitive workloads, which we configured through pod affinity rules - 2 of our 12 pods are in MedAI Labs' affinity group, accessible to other tenants only when MedAI Labs' queue is empty.

09 Architectural Diagrams: How the Pieces Connect#

One of the things I wish we had done earlier was draw clear architectural diagrams showing how all the isolation layers interact. When I was explaining our architecture to MedAI Labs' security team, I kept sketching the same three diagrams on a whiteboard. Here they are, formalized, so you do not have to recreate them from scratch.

Request Flow: From Client to Response#

Every request follows this path. Understanding the flow is critical for debugging latency issues, because each stage adds measurable time that shows up differently in our metrics.

The key insight from this diagram is that tenant identity is resolved at the API Gateway and tenant isolation is enforced at the EPP. The vLLM pods themselves are tenant-unaware. They process whatever requests the EPP sends them. This separation of concerns is deliberate: vLLM focuses on efficient inference, the EPP focuses on multi-tenant isolation. Neither needs to understand the other's internals.

Multi-Layer Isolation#

Our isolation is not a single wall. It is three concentric layers, each protecting against different failure modes. When we explained this to MedAI Labs' CISO, the layered model was what finally got us past the compliance review.

Capacity Borrowing Flow#

Capacity borrowing is one of the more subtle mechanisms in the system. When Acme Corp needs more capacity than their dedicated 4 pods provide, the EPP borrows from lower-priority tenants. This diagram shows the decision tree.

The borrowing cascade is strictly ordered by priority: P1 is throttled first, then P2, then P3. P4 (MedAI Labs) is never throttled for Acme Corp's benefit because MedAI Labs has their own proactive SLO enforcement. If MedAI Labs also needed capacity borrowing, their P4 requests would borrow from P1/P2/P3 before Acme Corp's P5 requests trigger preemption. The priority stack ensures that borrowing pressure flows downward, never upward.

10 Incident Retrospective: When CodeGen's Batch Job Starved the Production Chatbot#

This is the full story of the incident that changed how we build GPU infrastructure. I have written it as a detailed timeline because the details matter. Every minute of this incident taught us something, and I want you to learn from our pain instead of experiencing your own. This was the second of our three outages - the one that was most preventable and most expensive.

The Timeline#

The Detection Gap#

The incident started at 02:15 AM when KV cache crossed 80%, but the first alert did not fire until 02:47 AM - a 32-minute detection gap. During those 32 minutes, the cluster was sliding toward the cliff edge and nobody knew. Our KV cache alert threshold was set at 90%, and by the time it would have fired, the damage was already done. We now alert at 70% KV cache utilization with a rate-of-change trigger: if KV cache increases by more than 20 percentage points in 5 minutes, we page immediately regardless of the absolute level.

Root Cause Analysis#

The post-mortem identified five contributing causes, each of which was necessary for the incident to occur:

1. No per-tenant rate limiting: CodeGen Platform could submit unlimited requests because there was no rate limit configured. The token bucket algorithm would have capped their burst to 10 requests.
2. No per-tenant priority: The FIFO queue treated CodeGen Platform's batch work identically to Acme Corp's production API. WFQ with priority levels would have ensured Acme Corp's requests were served first.
3. No pod affinity: Acme Corp's "dedicated" pods were not actually dedicated. CodeGen Platform's requests were routed to all 12 pods, including the 4 we informally thought of as "Acme Corp's." Pod affinity rules would have reserved those pods.
4. Insufficient monitoring thresholds: The KV cache alert threshold of 90% was too high. By 90%, recovery is already impossible without killing requests. The rate-of-change alert we now run would have fired at 02:17 AM, 30 minutes before the first SLO breach.
5. No preemption capability: Even after detecting the problem, Sarah had no graceful way to prioritize Acme Corp's requests. She had to manually kill CodeGen Platform requests and block them at the gateway - a manual process that took 15 minutes.

The Fix: llm-d Configuration Changes#

Within 72 hours of the incident, we deployed llm-d's EPP with the tenant configuration shown in Section 3 of this post. The specific changes that would have prevented this incident:

• CodeGen Platform's token bucket rate limit (burstSize: 10, requestsPerSecond: 100) makes the 2,800 req/s burst physically impossible
• Priority levels (Acme Corp P5, CodeGen Platform P1) ensure Acme Corp's requests are always served before CodeGen Platform's
• Pod affinity (podAffinity: "dedicated" for Acme Corp) reserves 4 pods exclusively for Acme Corp traffic
• Proactive SLO enforcement begins borrowing capacity from CodeGen Platform when Acme Corp's p95 TTFT reaches 160ms (80% of 200ms threshold)
• Priority preemption allows the EPP to gracefully preempt CodeGen Platform requests instead of requiring manual intervention

Follow-Up: The Monthly Chaos Engineering Drill#

After this incident, we started running monthly "Noisy Neighbor" chaos engineering drills. On the first Wednesday of every month at 2 PM (during business hours, when the team is awake and ready), we simulate the exact conditions that caused INC-2026-0147:

1. A test script submits 5,000 CodeGen Platform-priority requests in a 30-second burst
2. Simultaneously, we inject synthetic Acme Corp traffic at 200 req/s (above their normal peak)
3. We verify that the EPP's rate limiter rejects the bulk of CodeGen Platform's burst (success criteria: fewer than 10 admitted)
4. We verify that Acme Corp's p95 TTFT stays below 200ms throughout the drill (success criteria: no breach)
5. We verify that preemption activates if needed and recovers Acme Corp within 12 seconds (success criteria: recovery under 15 seconds)
6. We verify that CodeGen Platform's rate-limited requests eventually complete within their 30-second SLO

Every drill has caught at least one issue: a Prometheus rule that stopped scraping after a config change, a PagerDuty routing rule that was pointing to a former employee's email, a pod affinity label that was accidentally removed during a Helm chart upgrade, and once, a regression in the EPP's preemption logic that would have allowed cascades deeper than our 2-level limit. The drill takes 20 minutes to run and about an hour to review results. It has prevented at least three incidents that we know of. Production will surprise you. Surprise it first.

11 Production Patterns We Wish We'd Known Earlier#

The isolation primitives described above are building blocks. Deploying them in production required operational patterns we learned through experience, some of it painful. If we were onboarding our five tenants again from scratch, here is exactly what we would do and in what order.

Tenant Onboarding Workflow#

When FinServ Team requested access to the shared inference cluster, we followed a structured onboarding workflow that we developed after the mistakes we made with Acme Corp and CodeGen Platform (where we skipped half these steps and paid for it with three outages):

Capacity Planning#

Sizing a shared cluster for 5 tenants with wildly different workload profiles requires understanding that not all tenants peak simultaneously. Our over-subscription ratio is 2.4x - the total allocated capacity across all tenants exceeds actual cluster capacity by 2.4 times. This is sustainable because:

• CodeGen Platform is elastic - it absorbs whatever capacity is available during off-peak hours and yields when Acme Corp's morning traffic ramps up
• Acme Corp has diurnal patterns - peak during US business hours (145 req/s), quiet overnight (12 req/s), which is when CodeGen Platform runs its heaviest batch jobs
• MedAI Labs' traffic is concentrated during clinical hours (8 AM to 6 PM) and drops to near-zero on weekends
• Research Squad's experiments are bursty but infrequent - typically 2 to 3 hours of heavy usage per day

The rule of thumb we developed: provision enough capacity to handle the simultaneous peak of all P5 and P4 tenants (Acme Corp + MedAI Labs), plus 50% of P3 tenants' average load (FinServ Team). Let P1 and P2 tenants (CodeGen Platform + Research Squad) fill the remainder. If Acme Corp and MedAI Labs alone would saturate the cluster at their peaks, you need more GPUs. We hit this threshold once and added 2 pods within 48 hours.

Burst Handling and Capacity Borrowing#

When Acme Corp experiences a traffic spike - as happened during their product launch when peak traffic hit 310 req/s, 60% above normal - the EPP automatically borrows capacity from CodeGen Platform and Research Squad. The mechanism is straightforward: lower-priority requests are throttled (their effective rate limit is temporarily reduced) to free pod capacity for Acme Corp. When the burst subsides, rate limits return to normal.

This capacity borrowing is invisible to Acme Corp - their requests are served normally within their p95 TTFT < 200ms SLO. CodeGen Platform sees increased latency and potentially 429 responses during the burst period. During Acme Corp's product launch, CodeGen Platform experienced a 23-minute window where 18% of their requests received 429 responses. Their batch job retried automatically and completed 35 minutes later than usual - well within their 30-second p95 e2e SLO and 95% availability target.

Graceful Degradation#

Eventually, every cluster hits its actual capacity limit. When even capacity borrowing and preemption cannot free enough resources, the EPP degrades gracefully in strict priority order:

1. CodeGen Platform (P1) is fully throttled (all requests receive 429) - their batch job queues locally and retries
2. Research Squad (P2) is throttled to 25% of their normal rate - experiments slow but do not fail
3. FinServ Team (P3) is throttled to 50% of their normal rate - reports take longer but generate
4. MedAI Labs (P4) and Acme Corp (P5) continue at full rate as long as physically possible

The EPP emits a llmd_epp_cluster_saturation metric and fires a PagerDuty alert when this state is entered. We have entered full degradation mode twice in 14 months of production operation - both times during Acme Corp traffic spikes that coincided with MedAI Labs' quarterly data processing runs.

12 Comparison with Other Approaches#

Before we deployed llm-d, we evaluated every alternative. We ran proof-of-concept deployments of three different approaches over six weeks. Here is what we found, and why we chose llm-d. I will be blunt: some of these alternatives could work for simpler setups, but none of them handled our five-tenant, mixed-workload, SLO-enforced scenario without significant custom engineering.

vLLM Native Multi-Tenancy#

vLLM is the inference engine llm-d uses as a backend, and it was our starting point. vLLM handles concurrent requests efficiently through continuous batching and can serve multiple LoRA adapters on the same base model. But vLLM does not provide tenant-level isolation primitives. There is no per-tenant priority, rate limiting, or SLO enforcement in the vLLM scheduler. All requests are treated equally. When we ran vLLM directly without a routing layer, CodeGen Platform's 10,000-request burst saturated the scheduler identically to the incident that started this whole journey. Good GPU utilization, zero tenant isolation.

NVIDIA Triton Inference Server#

Triton provides model-level isolation through its model repository architecture. It supports priority levels and rate limiting at the model level. But Triton's isolation is model-scoped, not tenant-scoped. When Acme Corp and CodeGen Platform both hit the same Llama 70B model, Triton does not distinguish between them. We would have needed to build an external gateway to map tenant identity to Triton scheduling parameters - effectively building the same routing layer that llm-d's EPP provides.

Custom Gateway Solutions#

We prototyped a custom rate limiter and priority router using Envoy with Lua plugins. It implemented per-tenant rate limiting and basic priority routing. But it was blind to GPU-level signals: it did not know about KV cache utilization at 94%, GPU memory pressure, or the continuous batching state of the inference engine. It could not make cache-aware routing decisions, could not perform graceful preemption (only hard kills), and could not proactively enforce Acme Corp's p95 TTFT < 200ms SLO based on actual GPU load. We scrapped it after two weeks.

KServe / Seldon Core#

KServe and Seldon Core are Kubernetes-native model serving platforms with model versioning, canary deployments, and autoscaling. Some multi-tenant awareness exists through namespace isolation and Istio-based traffic management. But neither understands LLM inference: they do not know about KV cache dynamics, continuous batching, or the autoregressive decode loop. Their autoscaling is based on generic metrics (CPU, memory, request rate) rather than LLM-specific signals like KV cache pressure or per-tenant latency percentiles.

Comparison Table#

The fundamental differentiator is that llm-d operates at the routing layer with full visibility into GPU-level signals. It knows about KV cache utilization, GPU memory pressure, continuous batching state, and per-tenant latency percentiles. This visibility is what enabled us to reduce Acme Corp's p95 TTFT from 2,400ms to 180ms during the CodeGen Platform incident, isolate 5 tenants across 12 GPU pods with per-tenant SLO compliance above 99.9% for our critical tier, and maintain cost attribution granular enough that every team lead knows exactly what they spend. Our custom Envoy gateway could rate limit, but it could not route based on KV cache pressure. Triton could isolate models, but it could not isolate Acme Corp from CodeGen Platform within the same Llama 70B deployment. vLLM provided the inference engine, but not the multi-tenant control plane.

13 What We'd Do Differently#

Fourteen months of running a shared GPU cluster for five tenants has given us a list of lessons that we wish we could send back in time to ourselves. Every one of these came from a real incident, a real conversation, or a real "why didn't we do this sooner" moment. If you are standing up a multi-tenant inference platform, start here. I have tried to make each lesson specific enough that you can act on it, not just nod along.