Slide 1 of 7

llm-d Open Source Framework

Multi-Tenant GPU Isolation

When multiple teams share inference GPUs, chaos follows. llm-d brings fair scheduling, SLA enforcement, and noisy neighbor protection to shared AI platforms.

5+

Tenants, One Platform

99.7%

SLA Compliance

0

Noisy Neighbor Breaches

Slide 2 of 7

Meet the Tenants

Five teams, five workloads, five different SLA tiers. All sharing the same inference platform.

A

Acme Corp

Platinum

Chatbot / p95 < 200ms
99.9% availability

B

MedAI Labs

Gold

Medical AI / p95 < 300ms
99.5% availability

C

FinServ Team

Silver

Financial / p95 < 400ms
99.0% availability

D

CodeGen

Bronze

Code gen / p95 < 500ms
95.0% availability

E

Research

Best Effort

Batch ML / p95 < 800ms
90.0% availability

Slide 3 of 7

The Noisy Neighbor Problem

When one tenant floods the shared GPU pool, every other tenant pays the price. Watch latency spike across the board.

400ms

Acme p95 (target: 200ms)

Burst

CodeGen floods the pool

SLO Breach

Cascading failures

Slide 4 of 7

Fair Scheduling with EPP

Token buckets, priority queues, and rate limiting. The Endpoint Picker Policy restores order to shared inference.

Token Buckets

Each tenant gets a bucket that refills at their rate limit. When empty, excess requests are throttled gracefully.

Priority Queues

Platinum requests jump ahead of Bronze. Higher SLA tiers always get first access to GPU resources.

Rate Limiting

No single tenant can consume more than their allocation, preventing noisy neighbor cascades.

Slide 5 of 7

GPU Isolation Architecture

From API gateway through tenant-aware routing, across isolated GPU pools with per-tenant SLA enforcement.

EPP Router

Tenant-aware request routing with SLA priority, token bucket enforcement, and least-load balancing.

Dedicated Pools

High-priority tenants get dedicated GPU pods. Lower tiers share a managed pool with fair scheduling.

SLA Monitor

Real-time compliance tracking with automatic scaling and rebalancing when SLOs are at risk.

Slide 6 of 7

Every Tenant Hits Their Target

With fair scheduling in place, SLA compliance rises across all tiers. The platform delivers on every promise.

99.9%

Acme Corp (Platinum)

99.5%

MedAI Labs (Gold)

99.0%

FinServ (Silver)

96.2%

CodeGen (Bronze)

93.8%

Research (Best Effort)

Slide 7 of 7

Deploy It Today

Three steps to multi-tenant GPU isolation. Install, configure tenant policies, and let llm-d enforce fairness.

1

Install with Helm

# Install llm-d
helm repo add llm-d \
  https://llm-d.github.io/llm-d/
helm install llm-d \
  llm-d/llm-d-inference-gateway

2

Configure Inference Pool

apiVersion: llm-d.ai/v1alpha1
kind: InferencePool
spec:
  targetPortNumber: 8000
  selector:
    app: vllm
  extensionRef:
    name: epp-multi-tenant

3

Define Tenant Policies

apiVersion: llm-d.ai/v1alpha1
kind: TenantPolicy
spec:
  priority: 5
  rateLimit: 80
  maxConcurrency: 25
  sloTarget: 200ms

4

Hybrid Isolation

spec:
  isolation:
    strategy: hybrid
    dedicatedPools:
      - acme-corp-platinum
    sharedPool:
      fairScheduling: true
      oversubscription: 1.5

View on GitHub Content Hub