llm-d Open Source Framework

One Base Model,
Many Talents

LoRA adapter routing in llm-d lets multiple fine-tuned models share a single GPU deployment -- slashing costs and eliminating resource fragmentation.

3.2x

Fewer Adapter Swaps

78%

Cache Hit Rate (EPP)

60%

GPU Cost Reduction

The Multi-Model Problem

Traditional deployments need one GPU per fine-tuned model. As teams multiply adapters for legal, medical, code review, and support -- costs spiral and GPUs sit idle.

⚠

Each model needs its own GPU
Dedicated hardware per fine-tune wastes resources

■

GPU utilization averages 20-40%
Most cycles sit idle between inference bursts

↑

Cost scales linearly with adapter count
Adding adapters means adding GPUs

✕

No resource sharing between models
Each deployment is an isolated silo

LoRA: Tiny Adapters, Massive Savings

Low-Rank Adaptation keeps the base model frozen and trains only tiny adapter matrices -- typically 2-5% of model size. Multiple adapters share one base model on a single GPU.

W = A × B where A is d × r, B is r × d, r « d

Base Model Frozen

The foundation weights stay unchanged. Only adapter matrices are trained and swapped.

2-5% Model Size

Each adapter adds minimal parameters. Store dozens of adapters alongside one base model.

Hot-Swappable

Adapters load and unload in milliseconds. No model reload, no cold start.

Smart Routing with EPP

The Endpoint Picker scores every pod for each incoming request -- balancing adapter affinity, KV cache warmth, queue depth, and GPU headroom to find the optimal target.

Score = affinity(0.50) + cache(0.20) + load(0.15) + gpu(0.15)

Adapter Affinity Routing

Routes requests to pods that already have the needed adapter loaded -- avoiding costly swap operations.

KV Cache Matching

Prefers pods with warm KV cache for the adapter, enabling faster time-to-first-token.

Load-Aware Balancing

Considers queue depth and GPU utilization to avoid overloading any single pod.

The Numbers Tell the Story

Benchmarked with 5 LoRA adapters on Llama 3.1 70B. Smart EPP routing vs naive round-robin across swap rate, latency, and cache efficiency.

3.2x

Fewer Swaps

58%

Lower P50 Latency

78%

Cache Hit Rate

Full System Architecture

From API gateway through the EPP router, requests flow to pods running vLLM with hot-swappable LoRA adapters. The routing sidecar reports adapter state back to EPP.

EPP Router

Pods with Adapters

Adapter Rings

Token Output

Deploy It Today

Enable LoRA routing in llm-d with a few lines of YAML. The EPP handles adapter-aware scoring automatically.

1

Base LoRA Deployment

apiVersion: inference.llm-d.ai/v1alpha1
kind: LLMInferenceService
spec:
  model: llama-3.1-70b
  adapters:
    maxSlots: 4

2

Enable EPP Routing

spec:
  routing:
    strategy: lora-affinity
    scoringWeights:
      adapterAffinity: 0.50
      kvCacheMatch: 0.20

3

Add Adapters

spec:
  adapters:
    - name: legal-assistant-v2
      source: huggingface/legal-lora
    - name: code-reviewer-xl
      source: huggingface/code-lora

4

Scale Out

spec:
  replicas: 6
  adapters:
    maxSlots: 4
    preloadStrategy: demand

View on GitHub → Content Hub →

One Base Model,Many Talents