From a Summit Stage to the CNCF: The Story of 1,000x

The room in Amsterdam went quiet for a beat too long.

It was March 24, 2026. KubeCon Europe. Thousands of engineers packed into a convention hall that smelled like fresh carpet and stale conference coffee. The CNCF Technical Oversight Committee had just announced the latest batch of Sandbox acceptances. When the slide landed on llm-d, donated jointly by IBM Research, Red Hat, and Google Cloud, the reaction was not polite applause. It was the kind of murmur that ripples through a room when people realize they are witnessing a turning point.

Ten months. That is how long it took llm-d to go from a conference announcement at Red Hat Summit to acceptance into the most important open-source foundation in cloud-native computing. Ten months from "here is an idea" to "this belongs alongside Kubernetes and Prometheus." For a project born at the intersection of AI and infrastructure, two fields known for hype and broken promises, the speed was almost disorienting.

But the people in that room knew why. They had felt the pain firsthand. They had watched their organizations bleed money on GPU clusters that sat half-idle because no one had built the intelligence layer between the models and the hardware. They had jury-rigged load balancers that did not understand KV caches, autoscalers that did not know a prefill from a decode, and monitoring dashboards that treated a 70-billion-parameter model the same as a REST API endpoint.

llm-d did not just solve a technical problem. It solved their problem. And in thirteen months, it went from solving it in theory to solving it at 50,000 tokens per second.

This is the story of how that happened.

The Problem Nobody Was Solving #

Here is something that drove infrastructure teams quietly insane throughout 2024 and early 2025: serving a single LLM on a single GPU was a solved problem. vLLM had cracked it. PagedAttention was a genuine breakthrough: elegant, effective, and widely adopted. If you needed to run Llama on one box and serve requests, you were in great shape.

But nobody needed to run one model on one box.

Real organizations needed to serve dozens of models across hundreds of GPUs, with predictable latency, controllable costs, and the kind of operational reliability that lets you sleep through the night when your inference fleet is processing ten thousand concurrent requests. And for that problem (the production problem, the at-scale problem, the problem that actually costs money) the industry had nothing.

Think about what a load balancer sees when it routes an LLM inference request. It sees an HTTP request. It picks a backend pod. Round-robin, least-connections, maybe something weighted. The request lands on whichever pod was "next." For a stateless web service, this is fine.

For LLM inference, it is catastrophic.

Every LLM request builds a KV cache: the model's working memory for that conversation. When a request arrives at a pod that already has the relevant prefix cached from a previous request, the pod can skip the entire prefill computation. That is the most expensive phase of inference: every matrix multiplication, every attention computation, every layer of the transformer. A KV cache hit on the right pod can cut time-to-first-token by 80% or more. Route that same request to a random pod, and you pay the full cost again from scratch.

This was not a load-balancing problem. It was a cache topology problem. And nobody's load balancer understood cache topology.

It got worse. LLM inference has two distinct computational phases that want completely different hardware. Prefill is compute-bound: it wants raw FLOPS, tensor cores, peak throughput. Decode is memory-bandwidth-bound: it generates one token at a time and reads the entire KV cache for each one. Running both on the same GPU is a compromise where neither phase gets what it needs. During prefill, the memory bandwidth sits idle. During decode, the compute units sit idle.

And then there was the multi-model problem. Enterprises do not serve one model. They serve Llama for conversation, CodeLlama for code generation, a Mistral variant for summarization, several LoRA adapters for domain-specific tasks, maybe DeepSeek for reasoning-heavy workloads. Each model has different hardware requirements, different scaling characteristics, different latency targets. LoRA adapters add yet another dimension: the same base model might serve dozens of adapters, and routing needs to know which adapters are loaded where.

All of this needed to compose with Kubernetes. Not alongside it, not instead of it, but with it. Organizations had invested years in their cloud-native infrastructure. Gateway API, Prometheus, HPA, service mesh. An inference framework that required its own parallel universe of custom tooling was dead on arrival.

Four insights. Four problems that were individually well-understood but had never been synthesized into a single, cohesive infrastructure project. Until May 2025.

The Day It Went Public #

Red Hat Summit. Boston. May 20, 2025. The announcement was remarkable not for what was announced but for who announced it together.

Red Hat, Google Cloud, IBM Research, CoreWeave, and NVIDIA stood on the same stage and said: we are building this together. These are companies that compete fiercely in the cloud and AI infrastructure markets. Their willingness to collaborate on a shared inference layer was the clearest possible signal that the problem was bigger than any one company's proprietary solution could address.

AMD, Cisco, Hugging Face, Intel, Lambda, and Mistral AI signed on as supporting organizations. UC Berkeley (the group behind vLLM and PagedAttention, led by Ion Stoica's lab) and the University of Chicago (the team behind LMCache and systematic prefix caching research) provided the academic depth that gave the project its technical soul.

The first tagged release, v0.1.0, shipped within days of the announcement. Forty-five tokens per second on a baseline configuration. The number was almost irrelevant. What mattered was that the architecture was sound, the routing decisions were correct, and the scaffolding for everything that would follow was in place.

The release cadence that would define the next thirteen months had begun.

The Chronicle: Month by Month #

Thirteen months is a long time in infrastructure. It is an eternity in AI. What follows is the story of llm-d told the way it actually unfolded: one month at a time, with all the momentum, the setbacks, the quiet breakthroughs that never made the press releases, and the loud ones that did.

The All-Star Roster #

Building infrastructure that the entire industry trusts requires more than good code. It requires the right people in the room. Look at who chose to put their name on llm-d, and what each brought to the table.

This was not a marketing exercise. Every one of these organizations contributed engineering hours, code, or research. When you see a roster like this aligned behind a single open-source project, it tells you something about the size of the problem being solved.

The 1,000x Rocket Ship #

Seven releases in thirteen months. Each one attacked the next bottleneck. Each one compounded on the last. The trajectory did not feel like incremental improvement. It felt like ignition.

v0.1 (May 2025): Ignition

The first release proved the architecture could work. The core EPP (Endpoint Picker) scoring algorithm shipped in its initial form: a weighted combination of KV cache affinity, queue depth, and pod health. Gateway API integration established the pattern that would carry through every subsequent release; llm-d would extend Kubernetes, not replace it. The health-check sidecar gave Kubernetes the signals it needed to manage inference pods correctly.

Forty-five tokens per second. Modest. But the routing decisions were correct, the control plane worked, and the architecture could scale. This was the "it compiles and routes correctly" release. The foundation was poured.

v0.2 "First Well-Lit Paths" (July 2025): The Thesis Proved

Two months later, v0.2 turned llm-d from an interesting architecture into a performance story. Three breakthroughs shipped simultaneously, each attacking a different bottleneck.

Prefix-cache-aware scheduling. The EPP now scored pods based on how much of the incoming request's prefix was already cached. A pod with 90% of the prefix cached scored dramatically higher than one starting cold. Sequential requests from the same user, or requests sharing system prompts and few-shot examples, consistently hit pods that could skip most of prefill. Cache hit rates above 80% became routine.

Prefill/decode disaggregation. Separate pod pools for each phase. Compute-optimized pods for prefill. Memory-optimized pods for decode. A new pd-sidecar container managed KV cache transfer between them. The handoff was invisible to the client: a request hit the EPP, got routed to a prefill pod, completed prefill, had its KV cache transferred to a decode pod, and started generating tokens. Two pods, one seamless experience.

MoE (Mixture of Experts) support arrived with DeepSeek-R1, including expert parallelism across GPUs. MoE models were becoming the dominant architecture for frontier models, and llm-d was ready.

The "Well-Lit Paths" name was deliberate philosophy. Each "path" was a tested, validated deployment pattern: single-node inference, disaggregated prefill/decode, multi-node MoE. Pick a path, not a pile of configuration knobs.

v0.3 "Wider Well-Lit Paths" (October 2025): Breaking Free

v0.3 was the release that proved llm-d was not just an NVIDIA story.

Cross-platform hardware support expanded to Intel XPU (Gaudi accelerators) and Google TPU alongside NVIDIA GPUs. This was a significant engineering effort: the EPP, the health-check sidecar, and the disaggregation logic all needed to work across hardware with fundamentally different memory architectures, compute characteristics, and driver stacks. But it was essential. Enterprises choosing llm-d needed to know they were not locked into a single vendor.

NIXL integration brought NVIDIA's Interconnect Library into the disaggregation story, enabling high-performance KV cache transfer using RDMA and NVLink where available. This was the piece that made disaggregation practical at production latency targets; without fast cache transfer, the overhead would have negated the benefit.

Wide expert parallelism across nodes pushed MoE serving past the single-node boundary. On H200 clusters, this delivered 2,200 tokens per second per GPU. An 11x improvement over v0.2.

The Inference Gateway reached v1.0 GA, providing a production-hardened ingress layer with TLS termination, rate limiting, and Gateway API protocol translation.

v0.4 (December 2025): The Latency Breakthrough

v0.4 went after latency. A 40% reduction in per-output-token latency for DeepSeek V3.1 on H200 hardware, driven by better scheduling, more efficient cache transfer, and improved decode batching.

Speculative decoding shipped: a smaller draft model generates candidate tokens, and the main model verifies them in parallel. When predictions are correct (which happens frequently for predictable text) multiple tokens get confirmed in a single forward pass. Integrating this into the distributed routing framework, where the draft and main models might live on different pods, required careful engineering.

CPU memory tiering for prefix cache offload meant that when GPU memory filled up, least-recently-used cache entries moved to CPU memory instead of being evicted. This preserved reusable cache state and significantly improved hit rates for workloads with many distinct prefixes. TPU and Intel XPU disaggregation support also expanded.

v0.5 "Sustaining Performance at Scale" (February 2026): Liftoff

This was the release that answered the only question that mattered: does this architecture actually work at scale?

The answer was emphatic.

Hierarchical KV offloading extended CPU tiering into a full three-tier hierarchy: GPU memory (hot), CPU memory (warm), disk (cold). This prevented "cache eviction storms," cascading cache misses when GPU memory fills up, by allowing graceful degradation instead of cliff-edge failures.

Cache-aware LoRA routing solved the multi-adapter problem. The EPP tracked which adapters were loaded on which pods, routing requests to pods with the correct adapter already in memory. At scale, this was the difference between meeting and missing latency SLAs.

Active-active high availability eliminated single points of failure. Multiple EPP instances running simultaneously, sharing state through lightweight coordination. Scale-to-zero autoscaling cut costs for bursty workloads: pods for infrequently-used models could scale to zero and spin up on demand.

v0.7 (June 2026): Maturity

After the performance fireworks of v0.5, the team made a deliberate choice: v0.7 was about operational maturity. CUDA 13.0.2 support. Standalone mode as the default deployment pattern. A documentation overhaul that treated docs as a first-class feature. CI coverage expansion. The project was optimizing for a new user's first hour, not just an expert's peak throughput: a sign that llm-d was transitioning from impressive technology to reliable infrastructure.

The Numbers: From 45 to 50,000 #

I could list the numbers in a sentence, but they deserve context. Each jump in this table came from a different insight, a different breakthrough, a different team pushing past a different bottleneck. The compounding effect is what makes this story extraordinary.

The 1,000x improvement did not come from a single eureka moment. It came from three forces compounding on each other:

Better routing (cache hits). Prefix-cache-aware scheduling eliminated redundant work. Every cache hit saved a full prefill computation. At 80%+ hit rates, this alone delivered 3-5x effective throughput improvement on prefill-heavy workloads.

Disaggregation (hardware-optimal workloads). Prefill on compute-optimized pods. Decode on memory-optimized pods. Each GPU spending its time doing what it does best, not sitting idle waiting for the other phase to finish.

Scale-out (more GPUs, efficiently). Wide expert parallelism plus NIXL's fast cache transfer meant that adding GPUs delivered near-linear throughput scaling instead of diminishing returns. The 16x16 B200 topology achieved roughly 3,100 tok/s per decode GPU: utilization efficiency remained high even at 32-GPU scale.

This is what makes distributed inference optimization qualitatively different from single-node work. Single-node optimization has a ceiling defined by the silicon. Distributed optimization has a much higher ceiling, but only if the coordination overhead is low enough that adding nodes actually helps. llm-d's architecture kept that overhead low enough to ride the curve.

The Breakthroughs That Mattered #

The version-by-version walkthrough tells you what shipped. This section tells you how the hardest problems were solved. These are the engineering stories behind the numbers, told with enough technical depth to be useful to the CNCF community members who will inherit and extend this work.

The EPP Scoring Algorithm: The Brain of the Operation

At the heart of llm-d sits the Endpoint Picker (EPP), and at the heart of the EPP sits a scoring algorithm that makes the routing decision for every incoming inference request. The algorithm computes a weighted score across three dimensions: KV cache affinity, queue depth, and pod health. The insight that made this work was treating the three dimensions as complementary signals rather than competing priorities.

KV cache affinity measures how much of the incoming request's prefix is already cached on a given pod. The EPP maintains a lightweight index of prefix hashes per pod, updated asynchronously via the health-check sidecar. When a request arrives, the EPP computes the prefix hash and scores each pod by match ratio. A pod with 95% prefix overlap scores dramatically higher than one with 10% overlap, because the former can skip nearly all of the prefill computation.

Queue depth acts as a load-balancing counterweight. A pod with perfect cache affinity but a queue 50 requests deep is worse than a pod with moderate affinity and an empty queue. The weighting between these two signals is configurable but ships with defaults that the team calibrated against production traffic patterns from enterprise pilots. The defaults work well for most workloads; the configurability exists for the edge cases where they do not.

Pod health is the safety net. Pods reporting degraded health (high memory pressure, GPU thermal throttling, pending eviction) get their scores penalized regardless of cache affinity or queue depth. The health signal comes from the sidecar, which has direct access to vLLM's internal metrics. This three-signal approach proved to be the right abstraction because it separates concerns cleanly: cache affinity optimizes for computational efficiency, queue depth optimizes for latency, and health optimizes for reliability. No single signal dominates, and the weights let operators tune the balance for their specific SLA requirements.

Prefix Cache Routing: Routing IS Cache Management

The deepest insight in llm-d's design is deceptively simple: in a distributed inference system, request routing and cache management are the same problem. Where you route a request determines what gets cached where. What is cached where determines where you should route the next request. The two are inextricably linked, and any system that treats them as separate concerns will leave performance on the table.

Traditional caching systems (CDNs, database caches, web caches) use consistent hashing or similar techniques to distribute cached content across nodes. These approaches work for static or slowly-changing content but break down for LLM inference because the "content" (the KV cache) is dynamically generated by the inference computation itself. You cannot pre-distribute it. You cannot replicate it cheaply. And the value of a cache entry depends not just on its existence but on its relationship to incoming requests.

llm-d's prefix cache routing solves this by making the EPP aware of both the request's content (specifically its prefix) and the cache state of every pod. The EPP does not just route to the least-loaded pod; it routes to the pod where the request will benefit most from existing cached state. This transforms the routing layer from a dumb load balancer into an active cache orchestrator. Each routing decision implicitly shapes the cache topology of the entire fleet, concentrating related requests on pods where their shared prefixes are already warm. Over time, the fleet self-organizes into a cache-optimal configuration without any explicit cache management logic. The routing decisions are the cache management.

This insight, that routing IS cache management, is what separates llm-d from every general-purpose load balancer and serving framework that has tried to bolt on LLM support. It cannot be added as an afterthought. It has to be the foundational design principle.

Prefill/Decode Disaggregation: Splitting What Should Never Have Been Together

The two phases of LLM inference, prefill and decode, have fundamentally different hardware requirements. Prefill is compute-bound: it processes the entire input prompt through the transformer in parallel, performing massive matrix multiplications that saturate the GPU's tensor cores. Decode is memory-bandwidth-bound: it generates one token at a time, reading the entire KV cache for each token but performing relatively little computation per read. Running both on the same GPU is like asking a sprinter and a marathon runner to share the same pair of shoes.

The idea of splitting them was not new. Academic papers had explored it. But making it work in a production Kubernetes environment, with transparent handoff, acceptable latency overhead, and no visible impact on the client, was a genuine engineering achievement. The pd-sidecar container is the key component. It runs alongside the vLLM engine in both prefill and decode pods, managing the KV cache handoff between them. When a prefill pod completes the prefill phase, the sidecar serializes the KV cache, transfers it to the decode pod's sidecar (using NIXL when available for high-performance transfer), and signals the decode pod's vLLM engine to begin token generation with the received cache state.

The non-obvious challenge was latency. Every millisecond of handoff latency adds directly to time-to-first-token. The team spent weeks optimizing the serialization format, the transfer protocol, and the decode pod's cache ingestion path. On NVIDIA hardware with NVLink or InfiniBand, the handoff adds less than 10ms of latency, well within the budget for production workloads. On commodity networking, the overhead is higher but still manageable for throughput-oriented workloads where absolute latency is less critical than aggregate tokens per second.

NIXL Integration: Microsecond Cache Transfer

NVIDIA's Interconnect Library (NIXL) provides a unified abstraction over the various high-performance interconnects available in GPU clusters: NVLink for intra-node GPU-to-GPU communication, NVSwitch for multi-GPU topologies, InfiniBand for inter-node communication, and RDMA for direct memory access across the network. For llm-d, NIXL solved a specific and critical problem: making KV cache transfer between pods fast enough that disaggregation's latency overhead did not negate its throughput benefit.

The engineering challenge was integrating NIXL into the pd-sidecar without creating a hard dependency on NVIDIA hardware. The sidecar needed to detect available interconnects at runtime, negotiate the fastest available transfer path between source and destination pods, and fall back gracefully to standard TCP when high-performance interconnects were not available. The integration also needed to handle the asymmetry inherent in real clusters, where some pod pairs have NVLink connectivity (same node) while others have only InfiniBand (different nodes) or TCP (different racks).

The solution was a transport abstraction layer in the sidecar that wraps NIXL's API and exposes a simple send/receive interface to the rest of the disaggregation pipeline. At startup, the sidecar probes available interconnects and builds a capability map. When a transfer is initiated, it selects the fastest available path. On clusters with full NIXL support, KV cache transfers for a Llama 70B model complete in under 5ms. On clusters without NIXL, the same transfer takes 50-100ms over TCP, still usable but noticeably slower. This performance gap is the primary reason enterprises invest in high-performance networking for their inference clusters, and NIXL integration gives them a direct return on that investment.

Hierarchical KV Offloading: The Three-Tier Cache That Saved Scale

As llm-d scaled to larger clusters and higher request volumes, a failure mode emerged that the team had anticipated but not yet solved: cache eviction storms. When GPU memory fills up, the inference engine must evict cached KV entries to make room for new requests. Under heavy load, this creates a cascading effect: evicted entries cause cache misses, which trigger full prefill computations, which consume GPU compute and memory, which cause more evictions. The system oscillates between periods of high cache hit rates and periods of near-zero hit rates, with throughput swinging wildly between them.

Hierarchical KV offloading solved this by creating a three-tier cache hierarchy. The first tier is GPU memory: fast, expensive, limited. The second tier is CPU memory: slower for inference computation but much larger and cheap enough to hold significantly more cache entries. The third tier is disk (NVMe SSD): the slowest tier but effectively unlimited in capacity. When GPU memory pressure rises, the least-recently-used cache entries are offloaded to CPU memory rather than evicted entirely. If CPU memory fills, entries move to disk. When a request arrives that matches an offloaded entry, the entry is promoted back to GPU memory, bypassing the full prefill computation.

The key insight was that promoting a cached entry from CPU memory to GPU memory is dramatically faster than recomputing it from scratch. For a Llama 70B model, a full prefill computation for a 4,096-token prompt takes roughly 200ms. Promoting the same KV cache from CPU memory takes approximately 15ms. From disk, roughly 40ms. Even the slowest tier is 5x faster than recomputation. This turned cache pressure from a cliff-edge failure (evict, recompute, thrash) into a graceful degradation (offload, promote when needed, maintain throughput). At scale, hierarchical offloading was the difference between the 50,000 tok/s number and a system that collapsed under its own cache pressure.

Scale-to-Zero: Sleeping Without Forgetting

Kubernetes' Horizontal Pod Autoscaler can scale pods to zero. But for LLM inference, scaling to zero means losing all cached KV state, which means that when the first request arrives after a scale-up, every prefill starts cold. For infrequently-used models (a LoRA adapter that handles customer-service queries during business hours, for example), this cold-start penalty is significant: the first batch of requests after scale-up experiences dramatically higher latency than subsequent requests.

llm-d's scale-to-zero implementation preserves cached state across scale-down/scale-up cycles. When a pod scales to zero, its KV cache is serialized and stored in a persistent volume claim. When the pod scales back up, the cache is restored from storage before the pod begins accepting requests. The pod "wakes up" with warm caches, as if it had never slept. The engineering subtlety is in cache invalidation: the stored cache must be validated against the model version and configuration at restoration time, because a model update during the sleep period would make the cached KV state invalid. The implementation handles this by storing a model fingerprint alongside the cache and performing a fast validation check during restoration. If the fingerprint does not match, the cache is discarded and the pod starts cold. This happens rarely in practice, since model updates are infrequent relative to scale-to-zero cycles.

From Single Company to Global Commons: The Governance Story #

There is a cynical reading of corporate open-source that goes like this: a company builds something useful, slaps an open-source license on it for marketing purposes, maintains effective control through contributor asymmetry and unwritten rules, and eventually re-licenses or acquires the community's contributions when the business model demands it. This pattern has played out enough times that experienced contributors are right to be skeptical.

llm-d was designed from day one to make that pattern structurally impossible. The governance story is worth telling in detail, because it is a case study in how to do multi-vendor open-source right.

The starting position was deliberate. llm-d began as a Red Hat-led initiative, with significant contributions from IBM Research. Red Hat could have open-sourced the project under a permissive license while retaining control of the governance structures, the maintainer appointments, and the roadmap. Many companies do exactly this. Instead, Red Hat chose to architect the project for eventual vendor-neutral governance from the first commit. The Apache 2.0 license was selected specifically because it is the most widely trusted permissive license in the enterprise open-source ecosystem. It allows commercial use, modification, and distribution without copyleft obligations, which removes the most common barrier to corporate contribution. Every partner organization's legal team could evaluate it in a single review cycle.

Multi-vendor contribution was not an aspiration; it was a requirement. By the time of the May 2025 announcement, engineers from Red Hat, IBM Research, Google Cloud, NVIDIA, and CoreWeave had all contributed code. Intel and AMD engineers followed within weeks. The founding partners were not just logos on a slide; they were commit authors in the git log. This was essential for the CNCF application, which requires evidence of contributor diversity, not just corporate endorsement.

The CNCF application process itself was rigorous. The project submitted a formal proposal to the Technical Oversight Committee in January 2026, detailing the project's scope, governance structure, contributor diversity, and fit within the CNCF landscape. The TOC conducted due diligence: reviewing the codebase, evaluating the governance documents, assessing the health of the contributor community, and determining whether the project addressed a genuine gap in the cloud-native ecosystem. The review process took approximately two months. TOC members asked pointed questions about vendor concentration, roadmap independence, and long-term sustainability. The answers were strong because the project had spent ten months building the evidence.

The joint donation structure was unusual and significant. IBM Research, Red Hat, and Google Cloud co-donated the project. Three competing companies simultaneously transferring intellectual property to a neutral foundation sent a clear signal: no single company owned this project, and no single company would own its future. The legal coordination required to execute a three-company simultaneous donation was substantial. Each company's IP counsel had to review the contribution, confirm that all donated code was free of encumbrances, and authorize the transfer under CNCF's IP policy. The fact that all three completed this process demonstrates executive-level commitment to vendor neutrality.

Vendor-neutral governance in practice means specific, enforceable structures. Maintainer selection is based on sustained contribution, not corporate affiliation. Any contributor who meets the criteria (consistent high-quality contributions, demonstrated technical judgment, community participation) can become a maintainer regardless of their employer. The roadmap process is public: proposals are submitted as GitHub issues, discussed in community meetings, and prioritized through a transparent voting process. No company has a veto. No company has a reserved seat. Contribution guidelines are documented, enforced, and the same for every contributor regardless of their organizational affiliation.

The comparison to other successful CNCF projects is instructive. Kubernetes was donated by Google. Envoy was donated by Lyft. Both projects successfully transitioned from single-company origins to genuinely multi-vendor communities, but the transition took years of deliberate governance work. llm-d benefited from studying these predecessors. The multi-company co-donation at launch, the Apache 2.0 license from day one, and the early establishment of public governance processes compressed a transition that typically takes two to three years into ten months. The CNCF's acceptance timeline reflected this: the TOC recognized that llm-d arrived with governance maturity that most Sandbox applicants develop after acceptance, not before.

This governance story matters not because governance is exciting (it is not) but because governance is what determines whether a project survives its second year. Technology draws contributors in. Governance is what keeps them.

Where llm-d Fits: The Stack Is Layering #

Let me say something that might sound counterintuitive: llm-d does not have competitors. It has neighbors.

The inference stack is not consolidating into one winner-take-all framework. It is layering: execution engines, programming frameworks, runtime orchestration, fleet routing, and general-purpose serving are settling into distinct strata. The winners will be projects that compose gracefully with the rest of the stack, not projects that try to swallow it whole. Here is where the major projects sit.

Standing on the Shoulders of Giants #

Great infrastructure is rarely invented from scratch. It is synthesized from research breakthroughs that came before. llm-d's technical innovations are built on a body of academic work that deserves recognition.

vLLM and PagedAttention (UC Berkeley)

The breakthrough that made modern LLM serving practical. Woosuk Kwon, Zhuohan Li, Sicheng Lin, and others in Ion Stoica's lab introduced PagedAttention: treating the KV cache like virtual memory, with paging. Instead of pre-allocating contiguous memory blocks (leading to severe fragmentation), PagedAttention allocates cache in small, non-contiguous blocks managed by a page table. Memory utilization jumped from 50-60% to 95%+. vLLM became the de facto standard for LLM serving. It is the engine running inside every llm-d inference pod, and the Berkeley team are founding partners, ensuring the engine and orchestration layer evolve together.

LMCache and Prefix Caching (University of Chicago)

The University of Chicago's LMCache project formalized systematic prefix caching for LLM inference. The insight is elegant: many LLM requests share common prefixes. Every customer-service request shares a system prompt. Every few-shot request shares the same examples. Every multi-turn conversation shares the entire history. By caching KV state for shared prefixes and reusing it across requests, you eliminate redundant computation. This research directly informed llm-d's cache-aware routing: the EPP knows which specific prefixes each pod has cached and scores them based on match ratio. The UChicago team are founding partners.

NIXL (NVIDIA)

NVIDIA's Interconnect Library provides high-performance memory transfer between GPUs, between nodes, and across interconnect fabrics (NVLink, InfiniBand, PCIe, RDMA). For llm-d, NIXL is the enabler of practical prefill/decode disaggregation. When a prefill pod hands off its KV cache to a decode pod, that transfer needs microsecond latency and multi-gigabyte-per-second bandwidth. NIXL delivers both, using whatever the fastest available interconnect is. Without it, disaggregation's latency penalty would negate its throughput benefit.

The Enterprise Story #

While the open-source project moved at open-source speed, the enterprise track kept pace with a discipline that reflected Red Hat's decades of experience shipping open-source software to organizations where downtime is measured in dollars per second.

May 2025: Red Hat AI Inference Server launches alongside the open-source announcement. Day-one enterprise commitment. The product was built on the same vLLM + llm-d stack, with certified hardware compatibility, long-term support, security patching, and enterprise SLAs.

October 2025: Red Hat AI 3 reaches GA. llm-d capabilities go generally available in the enterprise product, including MaaS (Model-as-a-Service) capabilities, a Llama Stack-based Unified API, and MCP (Model Context Protocol) adoption.

February 2026: Red Hat AI 3.3 ships with v0.5 capabilities: scale-to-zero, LoRA routing, hierarchical cache management. The two-to-three-month lag between upstream and enterprise held steady.

May 2026: Red Hat AI 3.4 at Red Hat Summit in Atlanta. Two noteworthy developments. First, Red Hat AI Inference became available on IBM Cloud as a managed service. Second, and this is the strategically important one, the product expanded beyond OpenShift to vanilla Kubernetes on CoreWeave and Azure. Red Hat was saying that llm-d-based inference was valuable enough to ship on non-Red Hat Kubernetes distributions. The inference layer had become more important than the platform.

The cadence tells the story: every major upstream release appeared in the enterprise product within two to three months. Same engineers contributing to both. Upstream leads, downstream follows. That is how you build trust with both the community and the enterprise customer.

Beyond the Numbers: What the Metrics Tell Us #

Raw metrics are easy to cite and easy to misread. A thousand GitHub stars could mean a thousand interested engineers or a thousand people who clicked a button and moved on. What matters is not the numbers themselves but the patterns they reveal. Here is what a deeper look at llm-d's community health metrics actually tells us about the project's trajectory.

Contributor Diversity: The Health of the Ecosystem

The single most important metric for a CNCF project's long-term health is contributor diversity. A project dominated by one organization, no matter how well-intentioned, is fragile. If that organization shifts priorities, the project withers. llm-d's contributor base spans thirteen-plus organizations, with no single company accounting for more than 40% of commits. Red Hat and IBM Research are the largest contributors, but Google Cloud, NVIDIA, Intel, CoreWeave, and independent contributors collectively account for the majority of code. This distribution was not accidental; it was cultivated through deliberate decisions about which contributions to prioritize, which features to accept from external contributors, and how to structure the review process to avoid bottlenecks at any single company's maintainers.

Geographic distribution matters too. Contributors span North America, Europe, and Asia, with a growing presence from academic contributors in Australia. The community calls rotate across time zones quarterly, ensuring no single geography is structurally disadvantaged. First-time contributors versus repeat contributors is the metric that distinguishes a project with buzz from a project with staying power. Roughly 35% of first-time contributors come back for a second PR, which compares favorably to CNCF projects of similar age. The team attributes this to a combination of fast PR review times, clear contribution guidelines, and the "good first issues" label that provides a structured onramp for newcomers.

Release Cadence: What Seven in Thirteen Means

Seven releases in thirteen months translates to one release roughly every eight weeks. For context, Kubernetes maintained a quarterly release cadence in its first two years. Envoy shipped roughly every six weeks. llm-d's pace is comparable to the fastest-shipping successful CNCF projects, which is remarkable given the project's hardware complexity (every release must be validated across NVIDIA, Intel, and TPU hardware) and the multi-vendor coordination required for each release's feature set.

The cadence also reveals something about the project's engineering maturity. Shipping every eight weeks requires CI that works, a release process that is automated, and a contributor base that can deliver features on a predictable schedule. Projects that ship irregularly or unpredictably signal process immaturity. llm-d's regularity signals the opposite.

PR Velocity and Issue Resolution

Pull request review times are a direct measure of how welcoming a project is to contributors. If your PR sits unreviewed for weeks, you stop contributing. llm-d's median first-review time of under 48 hours and median merge time of under one week for non-architectural changes are strong indicators of maintainer responsiveness. For architectural changes (new features, API modifications, cross-cutting concerns), review times are longer because they involve multi-maintainer review and community discussion, which is appropriate.

Issue first-response time of approximately 24 hours means that someone who reports a bug or asks a question gets a human response the same day or the next. Critical bugs are triaged same-day. This responsiveness is powered by a maintainer rotation system that distributes triage responsibility across organizations and time zones, preventing burnout and ensuring coverage.

Documentation as a Health Signal

The documentation contribution rate is an underappreciated metric for project health. Projects where only core developers write docs end up with documentation that is accurate but incomprehensible to newcomers. Projects where newcomers write docs end up with documentation that is accessible but sometimes inaccurate. The healthiest pattern is when both groups contribute, and llm-d has achieved that balance. The v0.7 documentation overhaul was a project-wide effort that involved core maintainers writing architecture documents and newer contributors writing getting-started guides and troubleshooting runbooks. The result is documentation that serves both audiences, which is exactly what a project targeting CNCF Incubating needs.

What We Are Building Next #

Seven releases in, llm-d has proven its core thesis. Cache-aware routing works. Prefill/decode disaggregation works. They work at scale. They work across hardware vendors. But I want to be honest with you: the problems that remain unsolved are at least as exciting as the ones we have solved. And some of them are genuinely hard.

Multi-Cluster Federation

Today, llm-d routes within a single Kubernetes cluster. Enterprises increasingly run inference across multiple clusters: different regions for latency, different providers for redundancy, on-premise for data sovereignty. Multi-cluster federation would give the EPP a fleet-wide view, routing to the best pod regardless of which cluster it lives in. This requires cross-cluster KV cache state sharing, which is a hard distributed systems problem. The kind of problem that gets researchers out of bed in the morning.

Agentic Inference Patterns

AI agents are becoming primary consumers of LLM inference, and their workload patterns are fundamentally different. Agents make chains of calls: tool calls, reasoning steps, multi-turn planning, where each call depends on the previous output. An agent's entire chain should ideally route to the same pod group to maximize KV cache reuse. The EPP needs to understand request chains, not just individual requests. This is new territory, and the patterns have not yet been established.

Cost-Aware Routing

Not every request needs the fastest hardware. A batch summarization job can tolerate higher latency in exchange for cheaper spot instances. Cost-aware routing would factor in hardware cost alongside cache affinity, queue depth, and latency targets, routing to the cheapest available hardware that still meets the SLA. For organizations spending millions on GPU compute, this is where the real savings live.

Multi-Modal Inference

How do you cache and route image embeddings alongside text KV caches? As models become multi-modal, the routing layer needs to handle heterogeneous cache types. This is largely unexplored territory.

Model-Aware Autoscaling

A 7B model and a 70B model have fundamentally different scaling curves. Model-aware autoscaling would encode these characteristics into scaling decisions, choosing the right number and size of pods for each model based on its architecture, hardware requirements, and observed load patterns.

Deeper Hardware Integration

NVIDIA B200, AMD MI300X, Intel Gaudi 3, Google TPU v6: each has different strengths. Hardware-aware scheduling that matches model architectures to silicon characteristics would let heterogeneous clusters serve each model on the hardware that suits it best.

Looking Forward: A Letter from the Maintainers #

Come Build This With Us #

Here is what I know after thirteen months.

The foundation is laid. Seven releases. A CNCF home. Performance that went from 45 tokens per second to 50,000. An architecture that has been validated by partners whose names carry weight in every boardroom and every server room on the planet. Enterprise GA status. Cross-platform hardware support. Active-active high availability. The technology works.

But technology was never the hard part.

The hard part is building a community that can sustain the pace. That can absorb new contributors without burning out the early ones. That can maintain governance rigor while moving at the speed the AI inference market demands. That can navigate the inevitable tensions when partners who also compete need to collaborate on a shared roadmap.

If you have felt the pain of running LLM inference at scale, the wasted GPU-hours, the unpredictable latency, the jury-rigged load balancers, you already understand why this project exists. If you have worked with Kubernetes and wished the inference layer spoke its language natively, you understand the architecture. If you have contributed to open-source infrastructure and know the satisfaction of building something that thousands of organizations rely on, you understand the opportunity.

The foundation is laid. The ecosystem is forming. The hardest and most interesting problems are still ahead of us.

Come build this with us.