Your LoRA Adapters Are Thrashing Your GPUs

01 What a LoRA Adapter Actually Is (and Why Size Matters for Routing)

Picture your production cluster. You're running Llama-3.1-70B on 8 H100 pods. Your platform team has fine-tuned 12 LoRA adapters: legal-assistant-v2 for contract analysis, medical-qa-hipaa for clinical decision support, code-reviewer-xl for pull request review, customer-support-es for Spanish-language support tickets, and 8 more. Each adapter modifies the base model's behavior for a specific task without touching the original weights.

Low-Rank Adaptation (LoRA), from Hu et al. (2021), decomposes weight updates into two small matrices instead of modifying the full weight matrix. The math is simple:

mathW' = W + B x A

where:
  W  = original frozen weight matrix  (d x k)
  B  = down-projection               (d x r)
  A  = up-projection                  (r x k)
  r  = rank (4 to 64)                r << min(d, k)

The original weights never change. During inference, the adapter adds BAx to the base model's output Wx. That's it. The adapter is just the B and A matrices -- nothing else gets stored or transferred.

Rank determines everything about routing cost

The rank r controls adapter size, and adapter size determines swap cost. For a 70B model with 8192-dimensional hidden states:

• Rank 4 (~15 MB): Lightweight tone/style adjustments. Swaps in under 2ms over PCIe Gen5. Your customer-support-es adapter might be this small -- it's mostly adjusting language patterns, not domain knowledge.
• Rank 16 (~50 MB): The production sweet spot. Your legal-assistant-v2 and medical-qa-hipaa adapters live here. Meaningful domain specialization. Swaps in 3-5ms from host memory.
• Rank 64 (~200 MB): Heavy-duty specialization. Your code-reviewer-xl adapter needs this -- reviewing code in 15 languages requires significant behavioral changes. Swaps take 12-20ms.
• Rank 128+ (~400 MB+): Rarely justified. At this point you're better off with full fine-tuning on selected layers. Swap cost starts to eat your latency budget.

Adapters modify attention -- and that makes KV caches adapter-specific

Most production adapters target the Q (query) and V (value) projection matrices in each transformer layer. Some, like code-reviewer-xl, also adapt K and O projections for higher quality, roughly doubling adapter size.

architectureTransformer Layer (with LoRA on Q, V)
================================================

Input x (hidden_dim=8192)
    |
    +---> Q = (W_q + B_q * A_q) * x     # adapted query
    +---> K =  W_k * x                   # original keys
    +---> V = (W_v + B_v * A_v) * x     # adapted values
    |
    +---> Attention(Q, K, V)
    |
    +---> Output = W_o * attn_result
    |
    +---> FFN layers (not adapted)
    |
Output

The critical implication: the KV cache produced during prefill is adapter-specific. Key and value tensors reflect the adapted weight matrices. If you use a different adapter during decode, the KV cache entries are inconsistent and the output is garbage. Both prefill and decode pods must have the same adapter loaded. We'll come back to this in Section 11.

02 LoRA Mathematics: Rank, Alpha, and Attention Weight Modification

Understanding LoRA at the mathematical level changes how you think about routing decisions. The rank and alpha parameters control adapter behavior in ways that directly affect swap cost, inference latency, and memory pressure. If you are running LoRA adapters in production, you need this mental model.

The full decomposition with scaling factor

The original LoRA paper presents a simplified view. The complete formulation includes a scaling factor that is critical for training stability and production behavior:

mathW' = W + (alpha / r) * B * A

where:
  W      = original frozen weight matrix     (d x k)
  B      = down-projection matrix            (d x r), initialized to zeros
  A      = up-projection matrix              (r x k), initialized with random Gaussian
  r      = rank (typically 4 to 64)          r << min(d, k)
  alpha  = scaling hyperparameter            controls adaptation magnitude
  alpha/r = effective scaling factor         determines how strongly the adapter modifies behavior

The scaling factor alpha/r is the key parameter most practitioners overlook. When alpha = r, the scaling factor is 1.0 and the adapter's raw output is added directly to the base model's output. When alpha = 2 * r, the adapter's effect is doubled. When alpha = r / 2, it is halved.

Why alpha and rank interact the way they do

Think of it this way: increasing rank gives the adapter more capacity to represent complex transformations. But higher rank also means more parameters participating in the addition, which can destabilize the base model's behavior. The alpha parameter compensates. In practice:

• alpha = rank (scaling factor 1.0): The most common production setting. The adapter contributes its learned modification at "full strength." This is what you should start with.
• alpha = 2 * rank (scaling factor 2.0): Amplifies the adapter's effect. Useful when the training data is small or the adapter needs to make a more dramatic behavioral change. Risk: can overshoot and produce inconsistent outputs when the base model's knowledge conflicts with the adapter's training.
• alpha = rank / 2 (scaling factor 0.5): Gentler adaptation. Useful for style transfer or tone adjustment where you want the base model's knowledge to dominate. Your customer-support-es adapter might use this -- it adjusts language patterns but should preserve the base model's domain understanding.
• alpha fixed at 16 while rank varies: A common shortcut in research papers. When rank goes from 4 to 64, the effective scaling factor changes from 4.0 to 0.25. This means higher-rank adapters have a proportionally weaker per-parameter contribution, which provides implicit regularization. Many practitioners adopt this convention without understanding why, and it works fine. But if you need precise control, set alpha = rank and adjust from there.

Which attention matrices to adapt -- and why Q+V is the sweet spot

A transformer layer has four projection matrices in its attention block: Q (query), K (key), V (value), and O (output). Each can be independently adapted with LoRA. The choice of which matrices to target has direct cost and quality implications:

adapter configurationsConfiguration          Adapted Matrices    Relative Size    Quality Impact
-------------------------------------------------------------------------------------
Q only                 W_q                 1x               Minimal for most tasks
Q + V                  W_q, W_v            2x               Best quality/cost ratio
Q + K + V              W_q, W_k, W_v       3x               Marginal improvement over Q+V
Q + K + V + O          W_q, W_k, W_v, W_o  4x               Diminishing returns
All (+ FFN)            All linear layers    6-8x             Research only; massive adapters

Q+V is the production sweet spot. The query projection controls what each token "looks for" in context, and the value projection controls what information gets propagated. Adapting these two matrices lets you reshape both the attention pattern and the information flow without touching the key projection (which determines what each token "advertises" about itself) or the output projection.

Adapting K changes how tokens are matched during attention -- this is valuable for tasks requiring fundamentally different retrieval patterns (like code review, where you need to match brackets and scope rather than semantic similarity). Your code-reviewer-xl adapter might adapt Q+K+V for this reason, which is why it is 4x larger than legal-assistant-v2 (Q+V only) and takes correspondingly longer to swap.

Concrete parameter counts for a 70B model

Let's calculate exactly how large a LoRA adapter is for Llama-3.1-70B:

parameter calculationLlama-3.1-70B architecture:
  hidden_dim (d)     = 8192
  num_layers         = 80
  num_attention_heads = 64
  head_dim           = 128
  GQA groups         = 8 (grouped query attention)

  Q projection size: 8192 x 8192 = 67.1M params per layer
  K projection size: 8192 x 1024 = 8.4M params per layer (GQA)
  V projection size: 8192 x 1024 = 8.4M params per layer (GQA)

Rank-16 adapter (Q + V only):
  Per Q layer: (8192 x 16) + (16 x 8192) = 262,144 params = 0.26M
  Per V layer: (8192 x 16) + (16 x 1024) = 147,456 params = 0.15M
  Per transformer layer: 0.41M params
  Total (80 layers): 32.8M params
  Memory (FP16): 32.8M x 2 bytes = 65.6 MB

Rank-64 adapter (Q + K + V):
  Per Q layer: (8192 x 64) + (64 x 8192) = 1,048,576 params = 1.05M
  Per K layer: (8192 x 64) + (64 x 1024) = 589,824 params = 0.59M
  Per V layer: (8192 x 64) + (64 x 1024) = 589,824 params = 0.59M
  Per transformer layer: 2.23M params
  Total (80 layers): 178.4M params
  Memory (FP16): 178.4M x 2 bytes = 356.8 MB

These numbers drive the swap cost model directly. A 65 MB adapter transfers over PCIe Gen5 (64 GB/s) in about 1ms. A 357 MB adapter takes 5.6ms. The gap widens further when you factor in memory allocation, fragmentation, and kernel warm-up.

Weight merging vs. keeping adapters separate

There are two strategies for applying LoRA at inference time:

• Merged weights: Pre-compute W' = W + (alpha/r) * B * A and store the result. The adapter disappears -- inference uses the merged matrix directly with zero overhead. Downside: you can only serve one adapter at a time. Swapping requires re-merging with a different adapter, which means touching the full weight matrix (140 GB for FP16 70B).
• Separate weights (additive): Keep B and A matrices separate. During inference, compute Wx + (alpha/r) * BAx in two steps. Overhead: 3-5% per forward pass. Upside: multiple adapters can coexist in GPU memory simultaneously, and swapping only touches the small adapter matrices.

For multi-tenant serving, always keep adapters separate. The 3-5% compute overhead is negligible compared to the latency of re-merging weights on every adapter switch. This is what vLLM, SGLang, and TensorRT-LLM all do.

Numerical precision: FP16 adapters on INT4 base models

Production deployments typically quantize the base model to INT4 (4-bit) or INT8 while keeping LoRA adapter weights in FP16 (16-bit). This is not a precision mismatch -- it is intentional.

The base model has billions of parameters and a well-conditioned loss landscape. Quantization to INT4 loses minimal quality because the redundancy across billions of weights provides error correction. The adapter has millions of parameters and captures fine-grained behavioral adjustments. Quantizing the adapter to INT4 causes measurable quality degradation because each parameter carries more "signal" relative to its neighbors.

The compute path handles this by dequantizing base model weights to FP16 on-the-fly during matrix multiplication, then adding the FP16 adapter contribution. The overhead of dequantization is absorbed by the memory-bandwidth-bound nature of inference -- the GPU spends most of its time waiting for data from HBM anyway.

03 The True Cost of an Adapter Swap

A swap is not "copy 50 MB to GPU memory." It's a five-phase process, and the worst phase has nothing to do with data transfer.

GPU memory is a zero-sum game

On a running vLLM instance, GPU HBM is carved up among four consumers. There's no slack:

• Base model weights: 70B in FP16 = 140 GB. INT4 (GPTQ/AWQ) = 35 GB. This is fixed.
• KV cache pool: 20-40% of remaining memory after model weights. Pre-allocated.
• Adapter weight slots: Space for loaded adapters. Each slot sized for the largest adapter the pod might serve.
• CUDA workspace: Scratch memory, activation tensors, NCCL buffers for multi-GPU. ~5 GB.

When medical-qa-hipaa (rank-16, 50 MB) gets evicted to make room for code-reviewer-xl (rank-64, 200 MB), the allocator has to coalesce free blocks. That fragmentation overhead adds 2-10ms -- before any data even moves.

The five phases of a swap

These numbers are for a rank-16 adapter (~50 MB) on A100/H100 with PCIe Gen4/Gen5:

latency breakdownPhase                      Duration       Notes
---------------------------------------------------------------
1. Batch drain              0 - 500ms      Wait for active batch to finish
2. Weight deallocation      ~5ms           Free GPU memory, update allocator
3. Weight transfer          10 - 50ms      PCIe/NVLink from host/NVMe
   - 50MB via PCIe Gen5:    ~3ms
   - 200MB via PCIe Gen5:   ~12ms
   - 50MB from NVMe:        ~10ms
   - 200MB from NVMe:       ~40ms
4. Initialization           ~20ms          Register weights, set up CUDA kernels
5. Warmup inference         50 - 100ms     Cold L2 cache, JIT compilation,
                                           cuBLAS autotuning on new shapes
---------------------------------------------------------------
Total (best case):          ~85ms          Adapter in CPU memory, no drain
Total (worst case):         ~670ms         Large adapter on NVMe, full drain

The convoy effect: how one swap kills your whole cluster

This is the pattern that pages your on-call at 3 AM:

1. Pod A gets a request for finance-analyst-v3, but has legal-assistant-v2 loaded. Starts draining its batch.
2. While Pod A drains, it can't accept requests. The load balancer sends Pod A's traffic to Pods B and C.
3. Pods B and C are now overloaded. Queue depths spike.
4. The redirected traffic includes requests for adapters not on B and C. More swaps trigger.
5. Multiple pods are now simultaneously draining and swapping. Cluster throughput collapses. GPUs show 90%+ utilization -- but they're doing memory management, not inference.

convoy effect timelineTime -->  0ms     100ms    200ms    300ms    400ms    500ms    600ms
Pod A:   [serving][drain..][swap....][warmup.......][serving........]
Pod B:   [serving][serving][drain..][swap....][warmup.......][serv..]
Pod C:   [serving][serving][serving][drain..][swap....][warmup.....]
                                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                                     Cluster throughput collapse zone
                                     All pods cycling through swaps

In the worst case, the convoy effect drops your cluster to 10-20% of theoretical throughput. The compute is there. It's just gated behind serialized adapter swaps.

This is the problem that cache-aware routing exists to solve.

04 EPP's Scoring Function: How the Math Works

The Endpoint Picker (EPP) is llm-d's routing layer, running as an ext-proc filter in the Envoy-based Gateway API data path. Every request gets scored against every available pod. The pod with the highest composite score wins.

Filter first, then score

Stage one eliminates pods that can't serve the request -- draining pods, pods at max queue depth, pods that don't support the model type. This cuts the candidate set by 30-50% with cheap boolean checks. Stage two runs the full multi-signal scoring on the survivors.

The scoring formula

scoring functionscore = 0.50 * adapter_affinity
     + 0.20 * kv_cache_affinity
     + 0.15 * (1 - queue_depth / max_queue_depth)
     + 0.15 * (1 - gpu_utilization / 100)

A concrete example: scoring medical-qa-hipaa

Request comes in for medical-qa-hipaa. Two candidate pods:

worked examplePod-02: has medical-qa-hipaa loaded, queue 3/8, GPU 45%, KV cache 0.6
  score = 0.50 * 1.0   +  0.20 * 0.60  +  0.15 * (1 - 3/8)  +  0.15 * (1 - 0.45)
        = 0.500        +  0.120        +  0.094               +  0.083
        = 0.797

Pod-05: does NOT have it, queue 0/8, GPU 12%, KV cache 0.0
  score = 0.50 * 0.0   +  0.20 * 0.00  +  0.15 * (1 - 0/8)  +  0.15 * (1 - 0.12)
        = 0.000        +  0.000        +  0.150               +  0.132
        = 0.282

Winner: Pod-02 (0.797 vs 0.282)

Pod-05 is completely idle. Zero queue, 12% GPU. But it scores 0.282 because it doesn't have the adapter. Pod-02 is busier -- 3 requests queued, GPU at 45% -- but it scores 0.797 because skipping the swap is worth more than skipping the queue.

Why adapter affinity gets 0.50: the cost asymmetry

This isn't arbitrary. It's a cost argument.

Routing to a pod with queue depth 6 instead of queue depth 2 adds 50-100ms of queueing delay. Routing to a pod without the adapter adds 85-670ms of swap latency plus the batch drain that delays every other request on that pod. The swap penalty is 2-10x larger than the load-balancing penalty in the common case. 50x larger in the worst case.

A 50% weight on adapter affinity is, if anything, conservative.

If you have more than 3 adapters, you need cache-aware routing. Period. With 3 adapters on an 8-pod cluster, random routing gives you a reasonable chance of hitting a pod with the right adapter. With 12 adapters, that probability drops below 30%. With 50 adapters, it's single digits. The economics only go one direction.

Adapter affinity is not binary

The EPP distinguishes four levels of affinity:

pseudocodefunc scoreAdapterAffinity(pod, requestedAdapter) float64 {
    if pod.HasActiveAdapter(requestedAdapter):
        return 1.0    // Best: loaded and actively serving
    if pod.HasCachedAdapter(requestedAdapter):
        return 0.8    // Good: in GPU memory, needs kernel activation
    if pod.HasAdapterInCPU(requestedAdapter):
        return 0.4    // Moderate: in host memory, needs PCIe transfer
    if pod.HasAdapterCapacity():
        return 0.3    // Acceptable: can load without eviction
    return 0.0        // Worst: must evict to make room
}

Natural clustering -- no configuration required

One of the best properties of affinity-aware routing: adapter clustering emerges without any static mapping. Nobody configures "adapter X goes to pod Y." The scoring function does it:

1. First request for finance-analyst-v3 goes to the least-loaded pod (no pod has affinity, so load signals dominate).
2. That pod loads the adapter and now has the highest affinity score for it.
3. Next request for finance-analyst-v3 routes to the same pod.
4. Over time, each pod accumulates a working set of 2-6 frequently-requested adapters.

This clustering is self-healing. If a pod restarts and loses its adapter state, the EPP reroutes that adapter's traffic to other pods that still have it loaded. As the restarted pod receives requests, it rebuilds its working set organically. No operator intervention.

05 Multi-Adapter Serving: More Than One Adapter Per GPU

A common misconception: each GPU pod holds one LoRA adapter at a time, making swaps inevitable. Wrong. Modern serving engines run multiple adapters simultaneously on the same GPU, batching requests across them.

Shared base, additive overlays

Base model weights load once and serve every request. LoRA adapters sit alongside them as small additive overlays in separate memory regions. They don't interfere with each other.

memory layoutGPU HBM (80 GB on H100)
=================================================
| Base Model Weights (INT4)    |  ~35 GB        |
|-------------------------------|----------------|
| Slot 1: legal-assistant-v2   |  ~50 MB (r=16) |
| Slot 2: medical-qa-hipaa     |  ~50 MB (r=16) |
| Slot 3: code-reviewer-xl     |  ~200 MB (r=64)|
| Slot 4: (empty)              |  reserved      |
|-------------------------------|----------------|
| KV Cache Pool                |  ~30 GB        |
|-------------------------------|----------------|
| CUDA Workspace               |  ~5 GB         |
=================================================

vLLM's multi-LoRA support batches requests targeting different adapters in the same inference batch. The base model's forward pass runs once for the whole batch. Then, for each adapted layer, the engine applies the correct adapter's BA matrices per-sequence. The overhead of multi-adapter batching is under 5% -- the adapter computation is tiny compared to the base model's weight matrices.

How many adapters fit?

memory budgetavailable = total_hbm - base_model - kv_cache - cuda_workspace

Example (H100 80GB, INT4 70B model):
  available = 80 - 35 - 30 - 5 = 10 GB
  rank-16 adapters (50 MB each):  ~200 slots
  rank-64 adapters (200 MB each): ~50 slots

In practice, vLLM caps simultaneously loaded adapters at 8-64 (configurable), limited by management overhead rather than raw memory. But even 8 slots is enough when paired with smart routing -- the EPP ensures those 8 slots contain the adapters your traffic actually needs.

Eviction: LRU by default, but routing makes it rare

When slots are full and a new adapter arrives, the serving engine evicts the least-recently-used adapter. The EPP's routing and the pod's eviction policy work as a two-tier system: routing minimizes evictions by sending requests to pods that already have the adapter. When evictions do happen (because no pod has the adapter, or affinity pods are saturated), LRU removes the least-disruptive one. Neither tier alone is sufficient. Together, they keep swap rates in single digits.

Pin your heavy hitters

For adapters that get 60-80% of total traffic, configure permanent residency. Pinned adapters load at pod startup and never get evicted. If legal-assistant-v2 and customer-support-es account for 70% of your requests, pin them on 4 pods. The EPP knows about pinning -- a pinned adapter gets a higher effective affinity score because there's zero eviction risk.

Pin your top 5 adapters. Make it part of your deployment checklist. Look at your request logs from the last 30 days, sort by adapter request count, and pin the top 5 across enough pods to handle their peak traffic. This single step eliminates 50-70% of all adapter swaps in a typical deployment. Ten minutes of configuration saves hours of incident response.

06 Adapter Lifecycle Management: Load, Evict, Survive

Having smart routing is necessary but not sufficient. You also need a coherent strategy for managing adapter lifecycles on each pod: when to load, when to evict, how to handle pressure spikes, and what to do when everything goes wrong simultaneously. If you have more than 3 adapters per pod, you need an eviction strategy. Period.

LRU vs LFU: the eviction strategy tradeoff

When a pod's adapter slots are full and a new adapter needs to load, something has to go. The two standard strategies are:

LRU (Least Recently Used): Evicts the adapter that was used longest ago. Simple, low overhead, and works well when traffic has strong temporal locality -- meaning users tend to make bursts of requests for the same adapter within a short window. This is the default in vLLM.

LFU (Least Frequently Used): Evicts the adapter with the fewest total requests. Better at protecting popular adapters from eviction by rare but recent requests. However, LFU has a cold-start problem: a newly loaded adapter has zero frequency, so it immediately becomes the eviction candidate. Pure LFU also has no forgetting mechanism -- an adapter that was popular last week but unused today keeps its high frequency count.

Neither is ideal in isolation. Here is when each fails:

failure modesLRU fails when:
  A batch job sends 200 requests for a cold adapter.
  This "touches" the cold adapter, making it "recently used."
  Meanwhile, your steadily popular legal-assistant-v2 hasn't been
  requested in the last 3 seconds (busy processing responses).
  LRU evicts legal-assistant-v2. Next request for it triggers a swap.
  You just evicted your most popular adapter for a batch job blip.

LFU fails when:
  quant-model-v7 was your most popular adapter 3 weeks ago
  during a quant research sprint. It accumulated 50,000 requests.
  It hasn't been used in 2 weeks. But its frequency count is still
  50,000. No other adapter can unseat it. You're burning a precious
  adapter slot on a ghost.

Time-weighted LRU: the practical solution

The approach that works best in production is time-weighted LRU with frequency decay. Each adapter gets a score that combines recency, frequency, and a decay factor:

pseudocoderetention_score = frequency_weight * freq_in_last_window
                + recency_weight * (1 / seconds_since_last_use)
                + pinned_bonus   * is_pinned

# Evict the adapter with the lowest retention_score
# freq_in_last_window uses a sliding window (e.g., last 10 minutes)
# This naturally "forgets" stale frequency counts

The sliding window is the key innovation. By only counting requests within the last 10 minutes (configurable), you get frequency sensitivity without the stale-count problem. An adapter that got 50,000 requests last week but none in the last 10 minutes scores zero on frequency. An adapter getting 5 requests per minute consistently scores high.

Adapter preloading: predicting what you will need

The best eviction is the one that never happens. Predictive preloading uses historical request patterns to load adapters before they are needed:

• Time-of-day patterns: customer-support-es peaks during CDMX business hours. Start preloading at 8:45 AM CST on pods in the western availability zone. By 9:00 AM, the adapter is hot and ready.
• Day-of-week patterns: code-reviewer-xl spikes Monday through Thursday but drops 80% on weekends. On Friday evening, deprioritize it for eviction. On Monday at 7 AM, preload aggressively.
• Pipeline triggers: When a document upload event fires, preload doc-classifier-v2 and the likely follow-up adapter (legal-extract-v3 or medical-extract-v1) on the same pod.
• Tenant login events: When a user from Tenant X authenticates, preload their tenant-specific adapter on 1-2 pods before they submit their first request.

The "adapter stampede" problem

This is the scenario that breaks naive implementations: 100 requests for a cold adapter arrive within a 2-second window. Without coordination:

1. The EPP routes the first request to the least-loaded pod (Pod A). Pod A starts loading the adapter.
2. Requests 2-100 arrive while Pod A is still loading (recall: loading takes 85-670ms). The EPP doesn't know Pod A is loading -- it just sees Pod A as busy.
3. The EPP routes requests 2-50 to Pod B, which also starts loading the same adapter. Now two pods are draining and swapping simultaneously.
4. Requests 51-100 go to Pod C. Three pods loading the same adapter. Cluster throughput collapses.

The solution is load coordination with request coalescing. When the EPP routes a request that will trigger a swap, it marks that pod as "loading adapter X" in its state. Subsequent requests for adapter X are queued behind the first one on the same pod, rather than fanning out to other pods. Once the adapter is loaded, all queued requests are served from the now-hot cache.

This is a classic thundering herd problem, and the classic solution applies: coalesce the stampede into a single load operation, then serve from cache.

Memory pressure and emergency eviction

Under sustained load, KV cache memory grows and can squeeze adapter slots. When GPU memory pressure exceeds 90%, the serving engine needs to make hard choices:

• Tier 1 (soft pressure, 85-90%): Stop accepting new adapter loads. Route all requests to pods that already have the needed adapter. If no pod has it, queue the request rather than forcing a load.
• Tier 2 (hard pressure, 90-95%): Emergency eviction of the lowest-scoring adapter. Aggressively shrink the KV cache pool if possible.
• Tier 3 (critical, 95%+): Reject new requests with a 503 and backpressure signal. Better to reject some requests cleanly than to OOM-kill the serving process and lose everything.

If your adapter residency time is under 5 minutes, you are thrashing. Fix it before adding more adapters. Either increase adapter slots (more GPU memory), reduce adapter count (consolidate adapters), or add more pods (spread the load).

07 Production Patterns That Actually Work

Adapter-aware routing is not set-and-forget. These patterns come from real deployments.

Preload by time of day

Adapter traffic is predictable. customer-support-es peaks during CDMX business hours (9 AM - 6 PM CST). code-reviewer-xl spikes when US developers start their day. A preloading controller uses historical request logs to load adapters before traffic arrives. Cold-start latency drops from 1.2 seconds to zero.

Hot/warm/cold tiering

• Hot (GPU HBM): Actively serving. Zero swap latency. 4-64 adapters per pod.
• Warm (CPU host memory): Not on GPU but in host RAM. Load in 3-20ms over PCIe Gen5. Hundreds of adapters per node.
• Cold (NVMe/network): On disk or object storage. Load in 10-200ms. Unlimited capacity.

The EPP knows which tier each adapter is in. A warm adapter on Pod A scores higher (affinity 0.4) than a cold adapter on Pod B (affinity 0.0), even if Pod B is less loaded.

Affinity groups for pipelines

Some adapters get called in sequence. A document processing pipeline invokes doc-classifier-v2 first, then legal-extract-v3 or medical-extract-v1 based on the result. If these adapters are on different pods, you pay inter-pod latency and risk triggering a swap on the second pod.

Affinity groups hint to the EPP that certain adapters should co-locate. Over time, pipeline-related adapters end up on the same pods.

What to monitor

• Swap rate (swaps/sec/pod): Healthy is below 0.1. Above 1.0 means something is wrong.
• Adapter residency time: Under 30 seconds = thrashing. Under 5 minutes = warning. If your adapter residency time is under 5 minutes, you're thrashing. Fix it before adding more adapters.
• Cache hit ratio: Target 90%+. Below 80% needs tuning.
• Swap queue depth: Consistently non-zero = swap latency is your bottleneck.
• Active adapter count vs. slot capacity: If active adapters exceed 80% of total cluster slots, you're one traffic shift away from a cascade.

08 War Story: The Day Our Adapter Cache Thrashed and Latency Went 10x

This is the incident that changed how we think about adapter capacity planning. It happened on a Thursday in March, and the root cause took four hours to identify despite every standard metric looking normal.

The setup: everything running smoothly

We had an 8-pod cluster running Llama-3.1-70B in INT4 on H100 GPUs. Each pod was configured with 4 adapter slots. We were serving 40 LoRA adapters across the cluster for a multi-tenant platform with about 30 enterprise customers. EPP routing had been live for three weeks. Cache hit rates sat comfortably above 92%. Adapter residency time averaged 45 minutes. Swap rate was 0.04 swaps/second/pod, which is well within the "healthy" range. P95 latency held steady at 220ms.

The dashboards were green. The on-call rotation was quiet. Everyone felt good about the system.

The trigger: an A/B test nobody told infrastructure about

Thursday morning at 10:15 AM, the ML team deployed an A/B test. They wanted to evaluate new training techniques, so they created an experimental variant for each of their top 20 adapters. Every customer's traffic was now split 50/50 between adapter-name-v2 and adapter-name-v2-experiment. The number of distinct adapters in active rotation jumped from 40 to 60 overnight.

The ML team deployed this through their standard CI/CD pipeline. No infrastructure ticket was filed. No capacity review was requested. From their perspective, they were just uploading some new adapter weight files and updating a routing config. Each experimental adapter was only 65 MB. What could go wrong?

The slow burn: symptoms that didn't look like symptoms

By noon, P95 latency had crept from 220ms to 310ms. Nobody noticed. Traffic volume was up 5% from the A/B test split, which seemed like a plausible explanation for the slight increase. By 1 PM, P95 hit 380ms. Still within SLA. A few users reported "slightly slower responses" in the feedback channel, but nothing actionable.

At 2:30 PM, the latency curve bent upward sharply. P95 hit 600ms. A senior engineer noticed and started investigating. By 3:00 PM, P95 crossed 1.2 seconds. By 3:15 PM, it hit 2.2 seconds. PagerDuty fired three alerts simultaneously: P95 SLA breach, error rate spike (timeouts), and queue depth critical on 6 of 8 pods.

What was happening under the hood

With 60 active adapters and only 4 slots per pod (32 slots total across 8 pods), the working set no longer fit. The math was brutal: 60 adapters divided across 32 slots meant each adapter could only maintain residency on an average of 0.53 pods. Before the A/B test, 40 adapters across 32 slots gave each adapter 0.8 pods on average, which was sufficient because traffic was heavily skewed toward the top 10 adapters.

The A/B test broke this equilibrium. Every request for a production adapter had a 50% chance of being followed by a request for that adapter's experimental variant. The experimental variant would evict a different production adapter. That production adapter's next request would evict something else. The eviction chain cascaded across the cluster.

eviction cascadeTimeline on Pod 3:
  14:31:02  Load legal-v2-experiment (evicts medical-qa-v2)
  14:31:04  Load medical-qa-v2 (evicts finance-v3)
  14:31:07  Load finance-v3-experiment (evicts legal-v2)
  14:31:09  Load legal-v2 (evicts customer-support-es)
  14:31:11  Load customer-support-es (evicts legal-v2-experiment)
  14:31:14  Load legal-v2-experiment (evicts medical-qa-v2)
  ... (cycling continuously, every adapter evicted within 30 seconds)

Adapter residency time: 45 minutes --> 90 seconds --> 40 seconds
Swap rate: 0.04/s/pod --> 0.8/s/pod --> 1.8/s/pod

At a residency time of 40 seconds, each adapter served only 3-4 requests before being evicted. Every swap drained the batch pipeline, blocking all other requests on that pod for 200-500ms. With 1.8 swaps per second per pod, the pods were spending more time swapping than serving.

Why the standard dashboards lied to us

GPU utilization was 94% across the cluster. The autoscaler was configured to scale when utilization exceeded 90%, but it did not trigger because we were already at our pod limit. More importantly, the autoscaler saw 94% utilization and concluded the GPUs were busy doing useful work. They were not. They were draining batches, deallocating adapter weights, transferring new weights over PCIe, and reinitializing CUDA kernels. Actual inference computation had dropped to roughly 15% of GPU time.

Memory utilization was normal. Network throughput was elevated but not alarming. Error rates were initially low because requests weren't failing, they were just slow. The system was perfectly healthy by every metric we had, except the ones we didn't have.

The diagnosis: four hours of looking in the wrong places

Hour 1: Checked hardware. NVLink status, PCIe error counters, ECC memory errors, thermal throttling. All clean. Checked vLLM logs for crash loops or OOM events. Nothing. Restarted one pod as a test. It came back and immediately started thrashing.

Hour 2: Added detailed request-level tracing. Saw that individual requests were spending 400-600ms in "waiting for adapter load" states. But we attributed this to "cold starts" and looked for why adapters weren't being cached. The EPP logs showed high cache hit rates, which was confusing. Turns out the hits were real, but the adapters were being evicted immediately after serving the request, so the "hit" was followed by a miss 30 seconds later.

Hour 3: Someone finally asked the right question: "How many distinct adapters are we serving right now?" We queried the EPP's adapter affinity state table and found 60 unique adapter IDs active in the last hour. Previous days showed 38-42. The ML team's Slack channel mentioned the A/B test. The pieces connected.

Hour 4: We built a quick dashboard correlating adapter load/evict events with per-request latency. The correlation was unmistakable. Every latency spike aligned with a burst of evictions. We computed adapter residency time for the first time and saw 40 seconds. The system was swapping adapters faster than it could serve requests.

The fix: three changes in 20 minutes

Change 1 (immediate): Increased adapter slots from 4 to 8 per pod by reducing the KV cache pool from 30 GB to 28 GB. The 4 additional rank-16 adapter slots needed about 260 MB. Reducing KV cache by 2 GB slightly lowered max concurrent sequences from ~180 to ~165, but the swap reduction more than compensated.

Change 2 (same hour): Pinned the top 10 production adapters (the non-experimental versions) across the cluster. Four pods received pinned assignments for the five highest-traffic adapters. These adapters became immune to eviction. The experimental variants could still compete for the remaining unpinned slots, but the core production adapters were protected.

Change 3 (next morning): Added three new monitoring signals that we should have had from day one:

• Adapter residency time (p50 and p95) with alerts at 5-minute and 1-minute thresholds
• Per-second swap rate per pod with an alert at 0.5 swaps/sec (our previous daily-aggregate metric was useless for detecting cascades)
• Distinct active adapter count vs. total cluster slot capacity -- a ratio above 0.8 triggers a warning

Lessons that are now part of our deployment checklist

09 How llm-d Compares to Everything Else

Other systems tackle pieces of the multi-LoRA problem. Here's where each fits and where llm-d's EPP is the only solution.

Punica introduced SGMV (Segmented Gather Matrix-Vector) kernels for efficient multi-adapter batching in a single GPU kernel launch. Makes individual pods faster at serving mixed-adapter batches. Does not help you route requests to the right pod.

S-LoRA applies paged memory management to adapter weights, the same trick vLLM uses for KV cache. Better memory utilization per node. Still no inter-pod routing.

dLoRA handles dynamic adapter loading and migration across GPUs within a single node. Closest to the EPP in spirit, but scoped to one multi-GPU server, not a Kubernetes cluster. Routes on adapter presence only -- no KV cache, queue depth, or GPU utilization signals.

The EPP is complementary to all of these. It operates at the cluster routing layer. The pods underneath can run Punica kernels, S-LoRA memory management, and dLoRA-style intra-node migration. In practice, vLLM already incorporates ideas from Punica and S-LoRA. The EPP adds the cluster-level intelligence that none of these provide.

10 What Happens When You Turn This On

Multi-tenant SaaS: 50 customers, 12 pods

A B2B platform provides AI customer support for 50+ enterprise customers. Each customer has a custom LoRA adapter trained on their product docs and brand voice. The platform runs 12 vLLM pods on a 70B base model (INT4), handling 2,000 requests/minute.

Before: round-robin routing

Each pod gets a uniform mix of all 50+ adapters. Each pod holds 8 adapters in GPU memory. 50 adapters across 8 slots = 84% of requests need a swap. During peak hours, the cluster burns through 1,600 swaps/minute. The convoy effect is brutal -- P95 spikes from 400ms to over 2 seconds during swap cascades.

After: EPP adapter-affinity routing

The top 10 customers (70% of traffic) get their adapters pinned on 4 dedicated pods. The remaining 40+ adapters distribute across 8 pods with LRU eviction.

50% throughput increase. Zero hardware added. The only change was routing requests to the right pods.

Code assistant: 4 languages, 6 pods

A dev tools company runs 4 specialized adapters: Python, JavaScript, Rust, Go. 6 H100 pods. Traffic follows developer work patterns -- strong temporal locality per user, mixed traffic at the cluster level.

With round-robin, each pod churns through all 4 adapters despite having plenty of slots. The warmup penalty after each swap adds 50-100ms. With EPP routing, Python and JavaScript (the two highest-volume) each get natural affinity to 2 pods. Rust and Go share the remaining 2. Swap rate drops from 45% to 7%. P99 improves from 1.1 seconds to 320ms.

Legal research: 12 jurisdictions, session stickiness

A legal tech platform hosts 12 jurisdiction-specific adapters (ca-jurisdiction-v4, ny-jurisdiction-v3, federal-law-v5, etc.). Each adapted for local case law, citation formats, and procedural terminology.

Legal research has strong session locality: a lawyer working on a California case makes dozens of sequential queries, all targeting ca-jurisdiction-v4. The deployment uses EPP with time-of-day preloading -- when the New York office opens at 9 AM ET, ny-jurisdiction-v3 and federal-law-v5 are preloaded onto pods near the eastern availability zone.

Cold-start latency for the first query of the day: 1.2 seconds to 180ms. Subsequent queries in the same session: consistently under 200ms because the EPP routes every request to the same pod.

11 Disaggregated Prefill/Decode with LoRA Adapters

llm-d supports prefill/decode disaggregation -- separating the two phases of LLM inference onto GPU pools optimized for each phase. Combined with adapter-aware routing, this creates unique challenges and compounding benefits.

Why disaggregate?

Prefill (processing the input prompt in parallel) is compute-bound -- high FLOPS, high arithmetic intensity. Decode (generating tokens one at a time) is memory-bandwidth-bound -- reads full model weights per token but does little computation per weight. When both phases run on the same GPU, they compete for resources inefficiently. Disaggregation lets you configure each pool for its workload.

Both pods need the right adapter

This is where it gets tricky. The prefill pod needs the adapter to compute correct KV cache entries (because LoRA modifies Q, K, V projections). The decode pod needs the same adapter to generate consistent tokens and extend the cache. A KV cache computed with legal-assistant-v2 is useless to a decode pod running medical-qa-hipaa.

The EPP handles this by maintaining two independent affinity maps -- one for prefill pods, one for decode pods. When a request arrives, the EPP scores a prefill pod by affinity and load, then pre-selects a decode pod with the same adapter before dispatching. When prefill completes, the KV cache ships to the pre-selected decode pod via RDMA/NVLink/TCP. Decode starts immediately with zero swap overhead.

The pre-selection step is critical. Without it, you'd need to find a decode pod with the right adapter after prefill finishes. Under load, that search could fail and force a swap at the worst possible time -- user already waiting, prefill cost already paid, KV cache sitting in memory consuming space.

The compounding benefit

Disaggregation alone improves throughput 30-60%. Adapter-aware routing alone cuts swap rates from 70%+ to under 10%. Together:

12 LoRA Routing vs. Deploying Separate Model Instances

The most common objection we hear: "Why not just deploy a separate model instance for each adapter? No routing complexity, no swaps, no cache management." Let's run the numbers on a realistic scenario and see why that approach collapses at scale.

The scenario: 12 adapters serving production traffic

You have 12 LoRA adapters, each serving an average of 200 requests per minute. Total cluster throughput requirement: 2,400 req/min. Base model: Llama-3.1-70B.

Option A: 12 separate model deployments

Each adapter gets its own dedicated model instance. No routing needed -- every instance only serves one adapter.

cost analysis - separate deploymentsMemory per instance:
  Base model (INT4):           35 GB
  KV cache:                    25 GB
  Single adapter:              0.05 GB
  CUDA workspace:              5 GB
  Total per instance:          ~65 GB  --> requires 1x H100-80GB

Instances needed:
  12 adapters x 1 GPU each  =  12 H100 GPUs

  But wait -- 200 req/min per adapter at 70B parameters with
  batch size 32 = ~600 req/min capacity per GPU.
  So 1 GPU per adapter is sufficient for throughput.

  For redundancy (N+1): 12 + 2 spare = 14 GPUs

Cost (on-demand H100 at $3.50/hr):
  14 GPUs x $3.50/hr x 730 hrs/month = $35,770/month

Option B: shared base model with LoRA routing

All adapters share the same base model across a smaller number of GPU pods. EPP routing ensures requests go to pods that already have the needed adapter loaded.

cost analysis - shared base with LoRAMemory per pod:
  Base model (INT4):           35 GB
  KV cache:                    30 GB (larger pool, shared across adapters)
  8 adapter slots x ~65 MB:    0.5 GB
  CUDA workspace:              5 GB
  Total per pod:               ~70.5 GB  --> fits on 1x H100-80GB

Pods needed:
  2,400 req/min total throughput
  Multi-adapter batching: ~800 req/min per pod (30% better than
  single-adapter due to improved GPU utilization from mixed batches)
  Minimum: ceil(2400/800) = 3 pods
  With headroom and redundancy: 5 pods

  5 pods hold 40 adapter slots (5 x 8). 12 adapters fit
  with 28 slots to spare for redundancy and hot spares.

Cost:
  5 GPUs x $3.50/hr x 730 hrs/month = $12,775/month

The comparison

Annual savings: $275,940. That is 64% cost reduction. The LoRA routing approach uses 64% fewer GPUs while serving the same traffic at comparable latency (slightly higher P99 due to the 8% swap rate, but within SLA for all but the most latency-sensitive workloads).

Break-even analysis: when does LoRA routing become mandatory?

At 1-2 adapters, the overhead of EPP routing and adapter management is not justified. Just deploy separate instances. The break-even point depends on your adapter count and traffic pattern:

• 3-5 adapters: LoRA routing saves 30-40% on GPU costs. Worth it if you have the engineering capacity to operate it.
• 6-15 adapters: LoRA routing saves 50-65%. Separate deployments are economically irrational at this scale. This is where most production deployments sit.
• 16-50 adapters: LoRA routing saves 70-85%. Separate deployments would require an absurd number of GPUs, most sitting idle at 15-20% utilization.
• 50+ adapters: Separate deployments are physically impossible at reasonable cost. LoRA routing with tiered storage (GPU/CPU/NVMe) is the only viable architecture.

13 What This Means for Your GPU Bill

Concrete math. No hand-waving.

calculationAssumptions:
  Requests per day:        50,000
  Avg swap penalty:        800ms (batch drain + transfer + warmup)
  Round-robin swap rate:   65%  (from case studies above)
  EPP swap rate:           8%   (from case studies above)

Round-robin waste:
  50,000 * 0.65 * 0.8s  = 26,000 GPU-seconds  = 7.2 GPU-hours/day

EPP waste:
  50,000 * 0.08 * 0.8s  = 3,200 GPU-seconds   = 0.9 GPU-hours/day

Daily savings:  6.3 GPU-hours
At $2.50/GPU-hr: $15.75/day = $5,750/year