llm-d Content Plan

5-minute deployment walkthrough of llm-d on OpenShift with GPU. Video covers the full flow from zero to working inference routing. Blog submitted for editorial review.

Data-driven benchmark of llm-d's KV-cache-aware routing vs round-robin on Llama 3.1 70B (FP8) across 3x H200 GPUs. 8 tests covering context scaling (500–32K tokens), concurrent load, agentic chains, and multi-turn conversations. Companion interactive web app visualizes results. Includes a “Cost Impact” tab translating TTFT and throughput gains into dollar savings — GPU-hours saved, cost-per-token reduction using real GPU pricing (H200, MI300X).

Interactive React app for conference showcases. Content adapts based on viewer's industry (17 sub-sectors across 7 categories) and role (5 levels from executive to platform engineer). 10 stages covering vertical optimization through expert parallelism, with animated architecture diagrams. Includes a “Decision-Maker Mode” — a shorter path that skips routing internals and focuses on business outcomes: “that cache hit saved X GPU-seconds.” Video tutorial walkthrough planned.

Deep dive into the routing algorithm. How the EPP tracks KV cache state across replicas and scores routing decisions. Why multi-turn conversations and agentic workflows benefit the most.

How to monitor llm-d inference workloads in production. Setting up dashboards for routing decisions, cache hit rates, GPU utilization, and latency. Ships a real Grafana dashboard.json on GitHub that people can import directly. The monitoring simulator web app becomes the companion guide.

Interactive visualization of inference costs as concurrent users scale from 10 to 1,000. Real GPU pricing (NVIDIA H200, AMD MI300X). Terrain-style visualization: calm rolling hill with llm-d vs. exponential cliff face without it. Companion blog translates benchmark numbers into dollar savings. 30-second screen recording for social media.

Advanced deployment guide showing separate prefill and decode workers. How to split compute-bound and memory-bound phases for maximum throughput.

Start with a working single-node inference setup, then crank up the load. Watch things break — visually. Queue depth grows, latency spikes, cache misses cascade, pods crash. Then “install” llm-d components one by one and watch each problem resolve. Companion blog: “I Tried to Scale LLM Inference Without llm-d. Here’s Where It Broke.”

Profile the llm-d coalition from a community perspective. CoreWeave, Google Cloud, IBM, NVIDIA, AMD, Microsoft, Oracle — why each showed up, what they’re contributing, and what it signals about the project’s trajectory. Written as a community advocate, not a partner press release. Companion video: short interviews or commentary on what open-source collaboration at this scale looks like.

How to serve multiple fine-tuned models from a single base model using llm-d's LoRA-aware routing. Practical guide for teams running multiple specialized models.

How enterprises serve multiple teams from shared GPU infrastructure using llm-d on OpenShift. Namespace isolation, per-team routing policies, quota management.

Interactive decision tree: answer 5 questions about your workload (model size, traffic pattern, hardware, latency requirements, multi-tenancy) and get a visual architecture diagram with a shareable deployment config. The kind of tool engineers bookmark and send to their team. Output includes Helm values and recommended llm-d settings.

Real-world deployment patterns with llm-d. Scaling from 2 GPUs to 100+. Operational lessons, monitoring, and what to watch for.

Visualize what an AI agent’s inference pattern looks like over time — bursty, multi-turn, tool-calling, context-switching. Show how naive round-robin routing handles it (cold starts everywhere, cache thrashing) vs. how llm-d routes it (sticky sessions, prefix cache hits). Air-traffic-control aesthetic. Companion blog: “I Built an AI Agent. Here’s the Inference Bill.” 2–3 minute video walkthrough.

How the open-source llm-d project becomes the Red Hat AI Inference Server inside Red Hat AI 3. Architecture mapping: what’s upstream, what’s packaged, what Red Hat adds (support, FIPS, lifecycle). The piece that connects community contributors to enterprise buyers — “the code you wrote is now running in production at [Fortune 500].”

How llm-d fits into the broader OpenShift AI platform. The full picture from model training to llm-d serving. Product updates and the integrated experience.

Visual cluster sandbox: see pods, send inference requests, watch the EPP route them in real time, see KV cache state update, watch responses stream back. Users send their own prompts. The booth demo that makes people crowd around. Evolved from the interactive demo into a live, explorable infrastructure visualization.

Open-source repo: Helm charts, deployment configs, and recommended settings for 3 scenarios (single-model serving, multi-model routing, agentic workloads). The “Deploy in 5 Minutes” blog becomes the README. Not just content — a tool that generates its own content through community adoption and contributions.

Landscape overview of where open-source LLM inference stands. Where llm-d fits, community growth, Red Hat's role in the ecosystem, and what's coming in 2027.

Distill.pub-style long-form technical explainer of llm-d’s routing algorithm with interactive inline figures. Scroll and diagrams animate, charts respond to input. Elevates the prefix-cache routing content to a format engineers share because it’s beautiful and educational. The “I can’t believe someone made this” piece.