Why Agentic Workloads Are the Killer App for KV Cache-Aware Routing

Agent Execution: Before and After llm-d

Say a customer support agent gets "I need to return the shoes I ordered last week." Here's what actually happens at the inference layer, step by step:

Why Agentic Workloads Are the Killer App for KV Cache-Aware Routing

Single-shot LLM calls are on their way out. The pattern that dominated 2023 and early 2024 (send a prompt, get a completion, done) is giving way to something way more demanding: agentic workloads. An agent doesn't make one LLM call per user request. It makes five. Ten. Sometimes thirty or more. Each call is a step in a reasoning chain: figure out what the user wants, pick a tool, parse the tool's output, decide the next action, write the final response, and sanity-check its own work along the way.

This is the shift that turns KV cache-aware routing from "nice to have" into "you really need this." With a simple chatbot, a cold cache miss on a single call adds maybe 400 milliseconds of extra prefill. Annoying, but fine. With an agent chaining eight LLM calls sequentially, that 400ms penalty compounds. You're not adding latency to one response — you're adding it to every step of the reasoning loop.

Here's a concrete example. A customer support agent gets: "I need to return the shoes I ordered last week." The execution looks something like: (1) parse the user intent, (2) call a tool to look up recent orders, (3) interpret the order data, (4) call a tool to check the return policy, (5) interpret the policy, (6) call a tool to initiate the return, (7) confirm the return was successful, (8) compose a final response to the user. That's eight LLM calls, and every one of them includes the same system prompt and tool definitions as its prefix. Without cache-aware routing, every single call lands on a random pod and pays the full prefill cost for the entire shared prefix.

The math is pretty damning. If each call carries roughly 2,300 tokens of shared prefix (a typical system prompt plus tool definitions), and your average prefill throughput is around 5,000 to 6,000 tokens per second on a well-utilized GPU, you're spending about 400ms just recomputing the KV cache for that prefix on every call. Multiply by eight calls: 3.2 seconds of pure waste. The user stares at a spinner for three extra seconds — not because the model is slow, but because the infrastructure keeps redoing work it already did.

This is exactly the problem llm-d solves. llm-d is a CNCF Sandbox project built for distributed LLM inference routing. It sits in front of your vLLM backends and uses its EPP (Endpoint Picker Policy) component to make routing decisions. When an agent's first call lands on a pod and computes the KV cache for its prefix, llm-d makes sure all subsequent calls from that agent go to the same pod. The prefix is already cached. Calls two through eight skip prefill entirely for the shared tokens and only compute attention for the new, call-specific tokens.

And the result is honestly kind of dramatic. Instead of 3.2 seconds of redundant prefill across eight calls, you pay 400ms once on the first call and roughly 65ms on each call after (for the small amount of new, call-specific context). Total prefill drops from 3.2 seconds to 855 milliseconds — a 73% reduction. That's not a micro-optimization. That's a different user experience.

The Prefix Cache Advantage: System Prompts and Tool Definitions as Shared Prefixes

To see why llm-d's approach works so well for agents, it helps to know how KV caches work at the GPU level and why prefixes are the natural unit of caching.

When a transformer processes a prompt, it computes key and value tensors for every token at every attention layer. These tensors are basically the model's "memory" of the tokens it has seen — they encode the contextual relationships between tokens that the model needs to generate output. For something like Llama 3.1 70B with 80 attention layers and a hidden dimension of 8192, each token produces a KV pair that eats a non-trivial amount of GPU memory and, more importantly, a non-trivial amount of compute to generate.

The key thing: KV cache entries are deterministic for a given prefix. If two requests share the same first 2,000 tokens, their KV cache entries for those 2,000 tokens are byte-for-byte identical. There's no randomness in the attention computation during prefill — the model computes the exact same key and value tensors no matter what comes after the prefix. So once you've computed the KV cache for a prefix on a specific GPU, any request that starts with that same prefix can reuse the cached tensors and skip straight to computing attention for the new tokens.

Why System Prompts and Tool Definitions Are Ideal for Caching

In agent architectures, the shared prefix isn't some small header or boilerplate. It's big, and it's the most expensive part of the prompt to process because it comes first and sets up the context window for everything after it.

A typical agent system prompt runs between 500 and 2,000 tokens. It defines the agent's persona, behavioral constraints, output formatting rules, and domain-specific instructions. This isn't a one-line "You are a helpful assistant." It's a detailed spec that might look like:

You are a senior customer support agent for Acme Corp.
You handle returns, exchanges, and order inquiries.

BEHAVIORAL RULES:
- Always verify the customer's identity before accessing order data
- Never disclose internal pricing or margin information
- Escalate to a human agent if the customer expresses frustration
  three or more times
- Always confirm actions before executing them
...
(continues for 800+ more tokens of guardrails, tone guidance,
 output format specifications, and escalation policies)

On top of the system prompt you've got the tool definitions — the structured schemas that tell the model what tools it can invoke, what parameters each takes, and what each returns. For an agent with five or six tools, these easily add another 200 to 1,000 tokens. Each definition has a name, description, parameter schemas with types and descriptions, required fields, and often example invocations or edge-case guidance.

The savings get even bigger with larger models. Prefill compute scales linearly with both token count and model size. A 2,300-token prefix on a 70B model takes roughly 400ms to prefill. On a 405B model, that same prefix might take over a second. For agents on frontier models, redundant prefix computation isn't just an annoyance — it's the dominant latency bottleneck.

llm-d's EPP tracks which pods have computed KV caches for which prefixes. When a new request comes in, the EPP looks at its prefix and checks whether any active pod already has it cached. If yes, route there. If not, pick a pod using load-aware heuristics and mark it as the new cache holder for that prefix. The lookup is fast (microseconds) because the EPP keeps an in-memory index of prefix hashes to pod assignments.

Multi-Turn Conversation Routing: Keeping Full Conversation Context Cached Across Turns

The prefix advantage goes way beyond the initial system prompt and tool definitions. In multi-turn conversations (which is basically every agentic workload), each turn adds to the prefix. Turn 1's prompt is the system prompt plus tool definitions plus the user's first message. Turn 2's prompt is all of that, plus the model's first response, plus the user's second message. By turn 5, the "prefix" shared with the previous turn might be 6,000 tokens or more.

This is where sticky routing really matters. llm-d supports session-aware routing that keeps an entire conversation pinned to the same pod across turns. The mechanics are simple: the first request in a conversation gets routed to a pod based on prefix matching or load balancing. llm-d then associates the conversation's session identifier with that pod. Every later request carrying the same session ID goes to the same pod, where the growing KV cache from previous turns is already sitting in GPU memory.

The numbers make the case on their own. Take a five-turn agent conversation where each turn involves two LLM calls (one to decide on an action, one to compose the response):

Without sticky routing, each of these calls could land on a different pod and pay full prefill cost every time. The later turns hurt even more, because the growing conversation history means more tokens to prefill from scratch. With llm-d, the first call eats the cold-start cost and every call after rides the growing cached prefix — later turns save the most.

One subtlety worth calling out. When a conversation's KV cache grows big enough, it can exceed what a single pod can keep resident in GPU memory alongside other active requests. llm-d handles this with cache eviction policies. If a conversation's cache gets evicted under memory pressure, llm-d can either re-route the conversation to a pod with more available cache memory, or eat the one-time recomputation cost and re-cache on the same pod. Because the EPP knows per-pod memory utilization, these decisions get made with full knowledge of cluster state instead of blindly sticking to a pod that no longer holds the cache.

Multi-Agent Cache Sharing: How Agents with the Same Tools Benefit from Shared Routing

So far this has all been about cache reuse within a single agent's execution or conversation. But the part that really got me is cache sharing across different agents.

In real platforms, it's common to have multiple agent types sharing overlapping tooling. A Research Agent, a Code Agent, and a Customer Support Agent might all have access to a web_search tool, a knowledge_base_query tool, and a send_notification tool. Their system prompts differ, but their tool definition blocks may be partially or fully identical. That's a cross-agent caching opportunity that traditional load balancers completely miss.

llm-d's prefix matching isn't all-or-nothing — it works at a granular level. If two agents share the first 800 tokens of their prompts (identical tool definitions) but diverge after that (different system prompts), llm-d can still route them to the same pod to take advantage of that 800-token shared prefix. The KV cache for those 800 tokens gets reused, and only the divergent portion is computed fresh. This partial prefix matching is one of the cleverer parts of llm-d's EPP implementation, and it's what makes multi-agent cache sharing actually practical.

This multi-agent cache sharing gets especially good on platforms doing the "tool marketplace" thing, where agents get composed from a library of standardized tools. The more standardized your tool definitions, the more cache sharing llm-d can exploit. Nice feedback loop there: standardizing common tool schemas doesn't just make developers' lives easier, it directly cuts inference costs through better cache utilization.

There's an architectural implication here worth sitting with. Traditional load balancers treat every request as independent and optimize for even distribution. llm-d inverts this: it deliberately creates affinity between requests that share prefixes, concentrating them on the same pods. Counterintuitive from a load-balancing perspective, but it's the right call for LLM inference — the compute savings from cache reuse far outweigh the marginal cost of slightly uneven load. The EPP balances both concerns, factoring in cache affinity and current pod load, and only routes to a cache-holding pod if it has enough capacity to serve the request without a lot of queuing delay.

MCP and llm-d: How Model Context Protocol Workloads Flow Through the Routing Layer

The Model Context Protocol (MCP) is quickly becoming the standard way to connect LLM agents to external tools and data sources. MCP defines a structured format for exposing tools, resources, and prompts to LLM apps. And there's an underappreciated consequence for inference routing here: MCP creates predictable, standardized prefix patterns that llm-d can exploit for caching.

When an MCP-compliant agent starts up, it connects to one or more MCP servers and pulls their tool manifests. Those manifests get translated into the tool definition blocks that form part of the agent's prompt prefix. Because MCP defines a standard schema for tool definitions (name, description, input schema with JSON Schema types, annotations), the resulting token sequences are highly regular and predictable across different agents connecting to the same MCP servers.

MCP Creates a Natural Caching Hierarchy

Look at the layers of an MCP-enabled agent's prompt:

1. Layer 1: MCP tool definitions, shared across all agents connecting to the same MCP servers. Highly cacheable.
2. Layer 2: Agent system prompt, shared across all instances of the same agent type. Cacheable per agent type.
3. Layer 3: Conversation history, unique per conversation session. Cacheable only with sticky routing.
4. Layer 4: Current user message, unique per request. Never cached.

llm-d can exploit every layer of this. Requests from different agent types that share MCP servers get Layer 1 caching. Requests from the same agent type get Layers 1 and 2. Requests within the same conversation get Layers 1, 2, and 3. The deeper the shared prefix, the bigger the savings.

MCP plus llm-d works out really well for tool-call-heavy workflows. Say an agent runs a multi-step task with five consecutive tool calls. Each one follows the same pattern: send a request to the LLM with the full context (system prompt + tool definitions + conversation so far + tool result), the LLM picks the next tool, repeat. Without cache-aware routing, each of those five requests recomputes the entire growing prefix from scratch. With llm-d's sticky routing, only the incrementally new tokens (the latest tool result and new user context) need fresh prefill.

MCP also has prompt templates: standardized prompt structures that MCP servers can expose for common tasks. When multiple agents use the same MCP prompt template, the token sequences share even more prefix overlap, which pushes llm-d's cache hit rates higher still. As MCP adoption grows and tool definitions get more standardized across the ecosystem, cache efficiency on llm-d-routed clusters should just keep improving.

One practical tip if you're the one running the platform. If you control the MCP servers your agents connect to, order your tool definitions consistently. llm-d matches prefixes from the start of the prompt, so putting the most commonly shared tool definitions first in the serialization order maximizes the shared prefix length across agents. Small detail, outsized impact on cache hit rates in multi-agent deployments.

Latency Analysis: How Much Time Intelligent Routing Saves Per Agent Execution

Okay, let's put real numbers on this. The analysis below models a realistic agentic workload and compares total Time-to-First-Token (TTFT) with and without llm-d's cache-aware routing.

Setup: A Typical Agent Execution

• System prompt: 1,500 tokens (behavioral rules, persona, output format)
• Tool definitions: 800 tokens (6 tools with JSON Schema parameters)
• Average user message + context per call: 150 tokens
• Total calls per execution: 8 (typical for a multi-step task)
• Model: 70B parameters, served on A100 80GB GPUs via vLLM
• Prefill throughput: ~5,500 tokens/second under moderate load

Without llm-d: Random Routing

With random or round-robin routing, each of the 8 calls lands on a different pod (or at best, a pod that already evicted any relevant cache). Every call pays full prefill for the shared prefix plus its unique context:

Call 1: (1500 + 800 + 150) / 5500 = 445ms
Call 2: (1500 + 800 + 150) / 5500 = 445ms
Call 3: (1500 + 800 + 200) / 5500 = 454ms  (growing context)
Call 4: (1500 + 800 + 250) / 5500 = 463ms
Call 5: (1500 + 800 + 300) / 5500 = 472ms
Call 6: (1500 + 800 + 350) / 5500 = 481ms
Call 7: (1500 + 800 + 400) / 5500 = 490ms
Call 8: (1500 + 800 + 450) / 5500 = 500ms

Average TTFT: ~469ms
Total prefill time across 8 calls: 3,750ms

With llm-d: Cache-Aware Routing

With llm-d, call 1 computes the full prefix and caches it. Calls 2 through 8 only compute attention for the new tokens (latest tool result, user message, or agent reasoning). Each call also picks up a small overhead (~40ms) for cache lookup and validation:

Call 1: (1500 + 800 + 150) / 5500 = 445ms  (cold start, full prefill)
Call 2: 150 / 5500 + 40ms overhead  =  67ms  (only new tokens)
Call 3: 200 / 5500 + 40ms overhead  =  76ms
Call 4: 250 / 5500 + 40ms overhead  =  85ms
Call 5: 300 / 5500 + 40ms overhead  =  95ms
Call 6: 350 / 5500 + 40ms overhead  = 104ms
Call 7: 400 / 5500 + 40ms overhead  = 113ms
Call 8: 450 / 5500 + 40ms overhead  = 122ms

Average TTFT (calls 2-8): ~95ms
Total prefill time across 8 calls: 1,107ms

At Scale: 1,000 Concurrent Agents

Now multiply by the number of concurrent agents in a real deployment. A mid-sized platform might run 1,000 concurrent agent executions, each making 8 LLM calls. That's 8,000 inference requests in a burst. Without llm-d, those requests eat 3,750 seconds of cumulative prefill time (across all requests). With llm-d, it drops to 1,107 seconds — 2,643 seconds of GPU compute freed up to serve actual new requests instead of recomputing cached prefixes.

In GPU-hour terms: at typical A100 utilization, the prefill savings across 1,000 concurrent agents come out to roughly 44 GPU-minutes of freed compute per minute of wall-clock operation. Nearly 44 GPUs' worth of prefill work, just gone. At current cloud pricing for A100 instances, that can mean tens of thousands of dollars a month in reduced infrastructure spend for a large agentic platform. And the savings grow linearly with concurrent agents and calls per execution.

One more dimension that matters in practice: tail latency. Without cache-aware routing, every call has roughly the same TTFT distribution because every call does roughly the same work. With llm-d, calls 2 through N have dramatically lower and more consistent TTFT, because cached prefill is a deterministic operation (cache lookup) rather than a variable one (GPU-dependent prefill computation). So llm-d doesn't just improve median latency — it tightens the p99, which is often the number that actually matters for user-facing agents.

Cost Impact: What Cache-Aware Routing Saves in Real Dollars

The latency wins translate straight into infrastructure cost savings. Freed GPU compute means you either serve more traffic on the same hardware or run the same traffic on fewer GPUs. Here's the math for a concrete deployment.

The Agentic Scaling Challenge: Why Agent Workloads Are Burstier and Harder to Predict

Agentic workloads don't just add more inference requests — they change the shape of traffic. Worth understanding if you're building infrastructure to serve agents at scale.

The Amplification Problem

In a traditional LLM app, one user request produces one inference call. Traffic is fairly predictable: 100 users averaging 2 requests per minute means roughly 200 inference calls per minute. You can provision and autoscale for that with standard techniques.

Agents break this completely. One user request might produce 3 inference calls (a simple lookup) or 25 (a complex multi-step research task). The amplification factor is variable and data-dependent — it depends on the agent's reasoning, the complexity of the request, and what the intermediate tool calls return. You can't predict it from the input alone. An agent that usually makes 5 calls might make 20 if a tool errors out and the agent decides to try alternatives.

Burst Patterns in Agentic Workloads

It gets worse: agents often generate parallel bursts of inference requests. A research agent that decides to search five sources at once fires off five inference calls simultaneously. A code agent that needs to analyze three files before picking a fix might kick off three parallel analysis calls. These bursts are sudden, short-lived, and intense.

Bursts also correlate in time. If your platform serves 100 concurrent users and each one triggers an agent, those 100 agents don't neatly stagger their calls. They cluster — lots of agents hit their "tool-calling" phase at the same time, and the parallel bursts stack. What you get looks less like a steady stream and more like sharp spikes separated by relative calm.

Traditional autoscaling just can't keep up with this. Kubernetes HPA (Horizontal Pod Autoscaler) typically works on a 15 to 60 second observation window, and spinning up a new vLLM pod takes 30 to 120 seconds (model loading alone can take 30+ seconds on large models, before GPU memory is even allocated and weights transferred). By the time the autoscaler notices the burst and provisions capacity, the burst is over. The new pods sit idle, costing money, while the next burst slams the existing pods that are still recovering from the last one.

How llm-d's Load-Aware Routing Helps Absorb Bursts

llm-d attacks the burst problem with load-aware routing that works at the request level, not the pod level. When a burst arrives, the EPP checks current queue depth and active request count on every pod in real time, then routes requests to pods with available capacity — spreading the burst across the cluster without waiting on autoscaling to provision anything.

The important bit: cache-awareness and load-awareness work together as one cost function, not competing priorities. The EPP doesn't blindly route every cache-sharing request to the same pod if that pod is slammed. It runs a dynamic cost model that weighs the compute savings of cache reuse against the queuing delay of routing to a busy pod. If the cache-holding pod's queue depth crosses a configurable threshold, the EPP routes the request to a less-loaded pod and takes the cache miss — the queuing delay would've cost more than the recomputation anyway.

This dynamic balancing matters most during bursts. When five parallel agent calls arrive at once, the first one or two might route to the cache-holding pod. But if calls three through five would queue behind them, llm-d sends them to other pods with capacity. Those calls pay more prefill, but they start executing immediately instead of sitting in a queue. Net result: lower end-to-end latency for the burst, even though individual cache hit rates drop.

llm-d also exposes metrics that external autoscalers can actually use. Per-pod queue depths, cache utilization ratios, request arrival rates — a much richer picture of cluster state than plain CPU or memory utilization. A custom autoscaler wired into llm-d's metrics can anticipate capacity needs from active agent sessions and their historical amplification factors, instead of reacting to pod resource utilization after the fact. For bursty agent traffic, predictive scaling like that beats reactive HPA rules by a lot.

Agentic Routing Challenges: Fan-Out, Cascading Latency, and Retry Storms

The routing layer for agentic workloads runs into three specific problems that don't exist (or are way milder) in single-call LLM apps. Worth knowing about all three before your agents get complex enough to hit them.

Fan-Out Patterns: The Thundering Herd Problem

A lot of agent architectures do fan-out: a single user request triggers N parallel LLM calls. Research and analysis agents do this constantly. Ask one to "summarize recent developments in quantum computing" and it might fan out five parallel searches — arxiv papers, industry news, GitHub activity, patent filings, conference proceedings. Each search generates its own LLM call to interpret and summarize results. All five run concurrently, so what looks like a single request to the user is a sudden 5x load spike at the inference layer.

That fan-out creates a thundering herd problem. If 100 users simultaneously trigger agents that each fan out to 5 parallel calls, the cluster suddenly faces 500 concurrent requests instead of 100. Traditional load balancers spread those 500 requests evenly across pods, which sounds reasonable — until you think about the KV cache. Without cache-aware routing, all 500 requests land on random pods. Even when requests share prefixes (common tool definitions, shared system prompts), the random distribution scatters those prefixes across different pods and kills any chance of cache reuse.

llm-d handles fan-out by noticing when multiple requests from the same agent session arrive within a short window (typically 50ms to 200ms) and routing them to a pod cluster that can share the cached prefix. The EPP tracks not just individual request prefixes but session-level prefix groups. When it detects a fan-out (same session ID, similar timestamps), it routes the requests to pods in the same availability zone or pod group, maximizing the odds they land on pods sharing the same prefix cache.

The scale impact adds up fast. In a deployment with 1,000 concurrent users where 20% trigger fan-outs averaging 5 parallel calls each, the instantaneous request load isn't 1,000 — it's 1,800 (800 single-call requests plus 1,000 fan-out calls from 200 users). Cache-aware routing lets those 1,000 fan-out calls share prefix caching, cutting the effective prefill load from 1,800 full prefills to roughly 1,000 (800 singles + 200 initial fan-out calls that cache the prefix). That's a 44% reduction in prefill compute during peak fan-out periods.

Cascading Latency: The Chain Is Only as Fast as Its Slowest Call

Fan-out gives you parallel load spikes, but plenty of agent workflows are the opposite: sequential dependencies, where one LLM call can't start until the previous one finishes. That's a cascading latency problem — total execution time is the sum of all sequential steps, and one slow call adds hundreds of milliseconds to the whole run.

Take a customer support agent handling a refund. The chain: (1) parse user intent (100ms generation), (2) look up order details via tool call (50ms), (3) LLM interprets order data and decides next step (120ms generation), (4) check refund eligibility via tool call (80ms), (5) LLM determines refund amount and policy (110ms generation), (6) initiate refund via tool call (200ms), (7) LLM confirms success and composes response (140ms generation). Seven steps, each depending on the one before.

Now add inference latency. Each of those four LLM calls (steps 1, 3, 5, 7) needs a prefill phase before generation starts. Without cache-aware routing, each call hits a random pod and pays full prefill: roughly 400ms per call for a 2,300-token prefix. Total: (400+100) + 50 + (400+120) + 80 + (400+110) + 200 + (400+140) = 2,400ms of LLM time plus 330ms of tool time = 2,730ms total.

With llm-d, the first call pays the 400ms prefill, but calls 2, 3, and 4 only pay 60ms to 80ms for their unique context (the prefix is cached). Total: (400+100) + 50 + (70+120) + 80 + (65+110) + 200 + (60+140) = 1,395ms of LLM time plus 330ms of tool time = 1,725ms total. A 1,005ms (37%) cut in end-to-end latency, purely from not redoing prefill on sequential calls.

llm-d's sticky routing exists precisely to stop these cascading penalties. Keeping an entire agent session pinned to the same pod (or pod group) means every call in a sequential chain rides the cached prefix computed on call one. The EPP tracks session IDs and holds pod affinity for the life of the session, typically via a TTL-based cache entry that expires after the session ends or after a configurable idle timeout (usually 5 to 15 minutes).

Retry Storms: When Failures Multiply Inference Load

Agents are brittle. Tool calls fail — timeouts, rate limits, transient network errors, malformed responses. When a tool call fails, most agent frameworks retry with a modified prompt, often adding context about the failure ("The previous call to search_database timed out. Try a more specific query"). That retry behavior can spiral into exponential retry storms where each failure generates more LLM calls, potentially 2x to 3x the original volume.

Concrete example. An agent calls a search tool with a broad query. It times out. The agent retries with a narrower query — also times out. So the agent splits the search into three parallel narrower queries (fan-out). Two succeed, one fails. The agent retries the failed one with an even more specific version. What started as one LLM call is now six (original + first retry + three parallel retries + one final retry). If 10% of agent requests hit this pattern, your cluster sees a 15% to 20% load increase from retries alone.

The interaction with routing is subtle but it matters. In a naive routing system, each retry lands on a random pod and recomputes the whole prefix from scratch. So a retry storm doesn't just bump call volume — it bumps prefill compute, because every retry is a cache miss. In the six-call example, that's six full prefills of the 2,300-token prefix: 2,400ms of redundant compute for what should have been one prefill plus five incremental updates.

Cache-aware routing contains retry storms in two ways. First, retries from the same agent session share the original call's prefix. If the original cached the prefix on pod A, llm-d routes the retry to pod A, where it's already in cache. The retry only pays to prefill the incremental context (the error message and modified query) — typically 50 to 150 tokens instead of 2,300. That cuts per-retry prefill cost by 85% to 95%.

Second, the EPP can do retry-aware load shedding. When a pod's queue depth crosses a critical threshold (overload territory), the EPP can tell the agent runtime to delay or abort retries, via HTTP 429 (Too Many Requests) responses with a Retry-After header. Agents that respect the signal back off, and the storm never overwhelms the cluster. It does require coordination between the routing layer (llm-d) and the agent runtime, but you end up with a system that degrades gracefully under retry load instead of falling over.

The pattern that works in practice: combine cache-aware routing (cheap retries when they do happen) with exponential backoff and jitter (spread retry load over time) and circuit breaking (fail fast when a tool is systematically down). llm-d handles the routing piece; the agent framework has to handle backoff and circuit breaking. Get all three working together and retry storms become an operational annoyance instead of a catastrophic failure mode.

Observability for Agents: Instrumenting Multi-Step Workflows with OpenTelemetry

Agentic workloads are opaque. A user submits a request and waits seconds for a response — but what happened during those seconds? Which LLM calls were made? Which tools ran? Where did the latency come from? Without instrumentation, debugging performance issues in a live agent is nearly impossible. This is where observability earns its keep, and OpenTelemetry (OTel) has become the standard for distributed tracing in agentic systems.

Instrumenting Multi-Step Agent Workflows with OpenTelemetry

OpenTelemetry gives you a vendor-neutral way to instrument code with traces (the flow of a request through a system) and spans (individual operations within that flow). For agents, the natural mapping: one trace per agent execution, one span per step — each LLM call, each tool invocation, each routing decision.

Here's what that looks like in practice. When an agent runtime gets a user request, it creates a parent span for the whole agent execution. Each LLM call gets a child span. Each tool invocation gets a child span. You end up with a hierarchical trace showing the complete flow of the agent's reasoning, including time spent in each phase.

import { trace } from '@opentelemetry/api';

async function executeAgent(userRequest) {
  const tracer = trace.getTracer('agent-runtime');
  const parentSpan = tracer.startSpan('agent.execute', {
    attributes: {
      'agent.type': 'customer-support',
      'user.request': userRequest
    }
  });

  try {
    // Step 1: Parse user intent
    const intentSpan = tracer.startSpan('agent.llm.call', {
      attributes: { 'llm.step': 'parse-intent' }
    });
    const intent = await callLLM(promptForIntent(userRequest));
    intentSpan.end();

    // Step 2: Tool call to look up order
    const toolSpan = tracer.startSpan('agent.tool.call', {
      attributes: { 'tool.name': 'lookup-order' }
    });
    const order = await lookupOrder(intent.orderId);
    toolSpan.end();

    // Step 3: LLM interprets order data
    const interpretSpan = tracer.startSpan('agent.llm.call', {
      attributes: { 'llm.step': 'interpret-order' }
    });
    const decision = await callLLM(promptForDecision(order));
    interpretSpan.end();

    // ... more steps ...

    parentSpan.setStatus({ code: SpanStatusCode.OK });
  } catch (error) {
    parentSpan.setStatus({
      code: SpanStatusCode.ERROR,
      message: error.message
    });
  } finally {
    parentSpan.end();
  }
}

That gives you a trace with a parent span and a bunch of child spans. Each span carries attributes (metadata) describing what operation it was, what parameters it used, and whether it succeeded. Export the trace to an observability backend (Jaeger, Tempo, Honeycomb) and you can see the entire agent execution as a waterfall — which steps ran sequentially, which ran in parallel, and where the time went.

Correlating Traces Across Agent Steps: Propagating Context Through the Routing Layer

Where OTel really pays off for agents is trace context propagation across service boundaries. When an agent runtime makes an LLM call to llm-d, which routes it to a vLLM backend, the trace context needs to survive all three hops (agent → llm-d → vLLM) so one Jaeger trace shows the complete flow, routing decisions included.

This happens via the traceparent HTTP header, part of the W3C Trace Context spec. The agent runtime includes a traceparent header (trace ID + parent span ID) on its inference request to llm-d. llm-d's routing layer extracts it, creates its own child span for the routing decision, and forwards the header (updated with its own span ID) to the vLLM backend, which creates a child span for the inference computation. Result: a three-level trace — agent execution → routing decision → inference computation.

llm-d preserves trace context with OpenTelemetry-aware middleware. When a request hits the EPP, the middleware extracts the traceparent header, creates a child span for the routing decision (with attributes like routing.strategy: cache-aware, routing.cache-hit: true, routing.pod: vllm-pod-3), and injects the updated trace context into the outbound request to vLLM. Happens transparently on every request, typically under 1ms of overhead.

The payoff is real. When an agent execution is slow, open the trace in Jaeger and you immediately see the bottleneck. Prefill? Cache miss. Generation? The model is grinding on the prompt. Tool call? Some external API is slow. Queuing delay at the routing layer? Cluster overload. It's all right there in the trace, no manual log correlation required.

Measuring End-to-End Agent Latency vs Individual Call Latency

One of the most common observability mistakes in agentic systems: conflating per-call TTFT (what vLLM reports) with end-to-end agent latency (what the user actually experiences). Different metrics, and the gap between them grows with agent complexity.

Per-call TTFT is the time from vLLM receiving a request to emitting the first token. It's the metric vLLM exposes on its /metrics endpoint and what most LLM serving benchmarks measure. For a single call, it's a decent proxy for user-perceived latency. For agent workflows, it isn't. An agent making eight LLM calls at an average TTFT of 150ms can still have 5 seconds of end-to-end latency, because the eight calls run sequentially with tool execution and network overhead in between.

The gap comes from sequential dependencies, tool execution time, and network round trips. Each LLM call waits on the previous tool call. Each tool call adds 50ms to 500ms depending on the external API. Each network round trip (agent → llm-d → vLLM → llm-d → agent) adds 5ms to 20ms depending on topology. It all compounds across the call chain.

The metrics I'd actually track for agent observability:

• Agent execution time: Total wall-clock time from user request to final response. This is what the user experiences. Measured at the agent runtime level.
• Total LLM time: Sum of TTFT + generation time across all LLM calls in the execution. Measured by summing span durations for all agent.llm.call spans.
• Total tool time: Sum of execution time across all tool calls. Measured by summing span durations for all agent.tool.call spans.
• Routing overhead: Time spent in llm-d's EPP making routing decisions. Measured by summing span durations for all routing.decision spans.
• Network overhead: Time spent in network I/O between services. Calculated as agent execution time - (total LLM time + total tool time + routing overhead).
• Cache hit rate: Percentage of LLM calls that hit the prefix cache. Measured by counting routing.cache-hit: true attributes in routing spans.

Track these over time and you catch trends per-call TTFT can't show you. Cache hit rate drops from 85% to 60%? End-to-end agent latency climbs even though per-call TTFT looks flat. External API degrades and tool time creeps up? End-to-end latency suffers even though LLM performance is unchanged. OTel's structured traces and span attributes let you compute all of this automatically from exported trace data.

Agentic Architecture Patterns: How llm-d Optimizes Common Agent Designs

Agentic AI has settled on a handful of common architectural patterns, and it's worth knowing how llm-d handles each one. The four below cover the majority of deployed agent systems.

ReAct Loops: Reason + Act Iterations

ReAct (Reasoning and Acting) is probably the most widely used agent architecture. The agent alternates between reasoning steps (deciding what to do next) and action steps (executing a tool call based on that reasoning), looping until it has enough information to produce a final answer.

A typical ReAct run: (1) the agent gets a user query, (2) generates a reasoning step explaining what it needs, (3) calls a tool to get it, (4) observes the result and reasons again based on what it learned, (5) either calls another tool or produces a final answer. Each iteration builds on the last, and the conversation history grows with every reasoning and action step.

The routing challenge: each iteration shares a growing prefix with the previous one. Iteration 1 starts with the system prompt (1,500 tokens) plus tool definitions (800 tokens) plus the user query (50 tokens) = 2,350 tokens. Iteration 2 adds iteration 1's reasoning (60 tokens) plus the tool result (20 tokens) = 2,430 tokens of prefix. Iteration 3 adds iteration 2's reasoning (70 tokens) plus the second tool result (40 tokens) = 2,540 tokens of prefix.

Without llm-d, each iteration lands on a random pod and recomputes the whole growing prefix from scratch. Iteration 1 pays 427ms for prefill. Iteration 2 pays 442ms. Iteration 3 pays 462ms. Total: 1,331ms. With llm-d's sticky routing, iteration 1 pays 427ms, but iterations 2 and 3 only pay for the tokens added since the last iteration — about 15ms for iteration 2 and 20ms for iteration 3. Total: 462ms. A 65% reduction.

llm-d's session-aware routing keeps the whole ReAct loop pinned to one pod. The first iteration caches the initial prefix (system prompt + tools + user query); every iteration after reuses that cache and only prefills the new reasoning and observation tokens. For a 10-iteration loop, that can take total prefill from over 4 seconds to under 600ms — an 85% cut that shows up directly in agent response time.

Tool-Augmented Generation (TAG): Pausing Mid-Generation for Tool Calls

Tool-Augmented Generation (TAG) is the pattern where the agent generates text, pauses to call a tool, folds the tool result into its context, and keeps generating. Unlike ReAct (discrete reasoning/action steps), TAG interleaves tool calls inside a single generation stream.

A TAG run might go: the agent starts generating ("Based on recent data..."), realizes mid-sentence it needs a specific data point, pauses, calls a tool, then continues with the result in context. The key difference from ReAct: it doesn't finish a full reasoning step before calling a tool — it calls the moment it realizes it needs to, even mid-sentence.

The inference challenge: each tool call breaks the generation stream. The agent generates 50 tokens, pauses for a tool call, then resumes. From the LLM's perspective that's two separate inference requests — one for the initial 50 tokens, one for the continuation after the tool result. The second request's prefix is the first request's full context plus the tool result.

Request 1 (initial generation):
  System prompt (1500 tok) + Tools (800 tok) + User query (40 tok)
  → Generate 50 tokens of response
  → Agent pauses, calls tool

Tool result returns (30 tokens of data)

Request 2 (continuation):
  System prompt (1500 tok) + Tools (800 tok) + User query (40 tok)
  + Generated so far (50 tok) + Tool result (30 tok)
  → Generate final 80 tokens of response

Without llm-d, request 2 lands on a random pod and recomputes the full 2,420-token prefix (system + tools + query + first 50 generated tokens + tool result). That's 440ms of prefill for a request that only adds 30 new tokens of context. With llm-d, request 2 goes back to the pod that handled request 1, where the KV cache for the first 2,340 tokens is still resident. Only the 80 new tokens (50 generated + 30 tool result) need prefill — about 15ms.

llm-d makes sure TAG continuations hit the cache by tracking request sequences within a session. When a request comes in with a session ID matching a recent request (within the last 5 to 30 seconds), the EPP routes it to the pod that handled the previous one, assuming that pod still holds the session's KV cache. This works even when the agent fires multiple parallel tool calls and then resumes — llm-d routes the resumption to the pod with the largest shared prefix, which is almost always the pod that handled the pre-pause generation.

Multi-Agent Orchestration: Supervisor Delegates to Worker Agents

In multi-agent orchestration, a supervisor agent delegates to specialized workers. The supervisor gets a user request, breaks it into subtasks, dispatches each to the right worker (research worker, code worker, analysis worker), then aggregates the results into a final response. Each worker has its own system prompt and tooling, but they may share common tool definitions (logging, notification, knowledge base access).

The routing angle: supervisor and workers all make LLM calls, and those calls may share partial prefixes. If everyone shares the same 5 common tools (600 tokens of tool definitions), cache-aware routing can exploit that shared prefix even though the system prompts differ.

Each worker might make 3 to 8 LLM calls. The supervisor makes 2 (initial task decomposition and final synthesis). That's 11 to 26 LLM calls for a single user request. Without cache-aware routing, each one lands on a random pod. With llm-d, supervisor and workers route to pods that can share the common tool definitions, and each worker's internal call sequence gets sticky routing on top.

llm-d's prefix matching gives multi-agent orchestration cross-agent cache sharing. When worker 1 and worker 2 both start, llm-d sees they share 600 tokens of common tool definitions and routes them to pods that have those cached. Worker 1's first call might cache the prefix on pod A. Worker 2's first call routes to pod A (or another pod in the same group) and reuses the cached tool definitions, saving 600 tokens of prefill. Even the supervisor's final synthesis call routes to a pod with the shared tool definitions cached, despite having a different system prompt from the workers.

At scale this adds up quickly. With 500 concurrent multi-agent requests (each spawning 3 workers), you've got 2,000 active agent sessions at any moment. Traditional routing spreads those across all available pods, fragmenting the cache. llm-d's cross-agent cache sharing concentrates agents with similar tool definitions onto the same pods, pushing cache hit rates from 20-30% (random routing) to 70-85% (cache-aware). At this scale, that's the difference between needing 20 pods and needing 12 for the same traffic.

Supervisor/Worker Patterns: Planning Agent Creates Subtasks for Specialist Agents

The supervisor/worker pattern is a specialization of multi-agent orchestration: the supervisor is a planning agent that builds a detailed execution plan with subtasks, and workers are specialists that execute them independently. The difference from generic orchestration is the hierarchy — the supervisor's system prompt is often a superset of the workers' prompts, or the workers' prompts come from templates that share structure with the supervisor's.

That hierarchy opens up prefix nesting. If the supervisor's system prompt is 2,000 tokens and includes a section defining the overall task domain, and each worker's system prompt is 1,200 tokens with the same domain definition (400 tokens) plus worker-specific instructions (800 tokens), then supervisor and workers share a 400-token prefix. llm-d's partial prefix matching can exploit that.

Concrete example: a supervisor agent gets "Build a web scraper for product data." Its system prompt defines the task domain ("You are building data extraction tools for e-commerce...") plus planning instructions. It creates subtasks: (1) design scraper architecture, (2) implement HTML parser, (3) implement rate limiting, (4) write tests, and dispatches each to a worker. The workers' system prompts start with the same domain definition (400 tokens) and then add worker-specific instructions.

Without llm-d, supervisor and workers land on random pods, and the domain definition (400 tokens) gets computed 5 times — once for the supervisor, once per worker. With llm-d, the supervisor's first call caches it, and the 4 workers route to pods with that prefix cached, saving 1,600 tokens of redundant prefill (400 tokens × 4 workers). If each worker makes 3 LLM calls, that's 12 calls riding the shared prefix cache — 4,800 tokens of prefill saved.

The pattern that works in practice: tag supervisor and worker requests with metadata about their relationship (supervisor ID, worker ID, task ID). The EPP uses it to route workers to pods that handled the supervisor's initial planning call, maximizing prefix sharing. Your agent framework has to put the metadata in the request headers, but the routing savings easily justify that small integration cost.

Pattern Comparison at a Glance

The four patterns above each behave differently in terms of LLM call volume, cache behavior, and how much llm-d's routing helps them. Here's a side-by-side to help you figure out which fits your workload.

Which Pattern Should I Use?

Picking an architecture comes down to task complexity, latency requirements, and how your agent talks to external tools. The decision tree below is a decent starting point.

Production Patterns for Agentic Inference

Running llm-d for agent workloads in production takes more than flipping on cache-aware routing. The patterns below are the kind of stuff you usually learn the hard way, and they apply whether you're serving dozens of concurrent agents or thousands.

Connection Pooling for Agent-to-Inference Communication

Agents are chatty. An agent making 8+ inference calls per execution shouldn't be opening and closing HTTP connections for each one. You want connection pooling between the agent runtime and the llm-d ingress — without it, TCP handshake and TLS negotiation add 5 to 20ms per call, and that compounds across the call chain just like prefill latency does.

The pattern: keep a pool of persistent HTTP/2 connections between each agent runtime pod and the llm-d routing layer. HTTP/2 multiplexing lets multiple inference requests fly concurrently over one connection, which matters a lot for agents doing parallel tool calls. Size the pool from the max concurrent inference requests per agent runtime pod — typically 10 to 50 connections for a pod running 20 to 100 concurrent agent executions. Watch connection utilization to right-size it: too few and you get head-of-line blocking on parallel bursts; too many and you're wasting kernel resources on idle sockets.

Request Coalescing: Batching Shared-Prefix Calls

When multiple agents share a prefix and fire inference requests at roughly the same time, llm-d can coalesce them into a single prefill computation. Instead of computing the shared prefix's KV cache once per request, the router spots matching prefixes within a short window (typically 10 to 50ms) and batches their prefill on the same pod. Each request still gets its own completion, but the shared prefix gets computed once and the KV cache is shared across the whole batch.

Coalescing shines on multi-tenant platforms where lots of agents share tool definitions. If 20 requests with the same 800-token tool definition prefix arrive within a 30ms window, coalescing cuts their prefill compute by 95% — prefix computed once, and the 20 unique suffixes batched for efficient GPU utilization. Conceptually it's like vLLM's continuous batching at the generation level, extended by llm-d to the routing and prefill layers.

Priority Queues: Differentiating Interactive and Background Calls

Not every call in an agent execution is equally urgent. The final response to the user is latency-sensitive — someone's actively waiting on it. Intermediate tool-call decisions, background reasoning steps, and self-verification checks, less so. Tag your inference requests with priority levels and let llm-d route accordingly.

The EPP can fold request priority into its routing cost function. High-priority requests weight queue depth more heavily (waiting is unacceptable), low-priority requests weight cache affinity more heavily (recomputation is wasteful when nobody's waiting). That way the cluster serves latency-sensitive interactive requests fast while still squeezing max cache utilization out of background work that can tolerate the wait.

Circuit Breaking: Handling Cache Thrashing

Cache thrashing happens when a pod's KV cache keeps getting evicted and recomputed under memory pressure from too many concurrent sessions. It's a nasty feedback loop: llm-d routes requests to the pod because it supposedly "has" the cache, but the cache got evicted before the request lands, so the request pays full prefill plus the queuing delay of being pinned to a specific pod instead of the least-loaded one. Worst of both worlds — cache-miss latency with affinity-routing overhead.

The fix is circuit breaking at the cache level. llm-d watches each pod's cache hit rate in real time. When a pod drops below a configurable threshold (say, under 40% over the last 60 seconds), the EPP marks it "cache-degraded" and stops sending cache-affinity requests its way, falling back to pure load-based routing until the hit rate recovers past a recovery threshold (say, 60%). That breaks the feedback loop and makes sure cache-aware routing only kicks in when it's actually paying off.

This does require the vLLM backends to expose cache statistics to the EPP — cache hits vs misses over a rolling window. llm-d's health-check protocol includes those metrics, so the EPP can make circuit-breaking decisions without adding latency to the hot path.

Graceful Degradation: When All Pods Are Cold

Sometimes cache-aware routing has nothing to work with: cluster cold starts after a deploy, scaling events bringing up new pods with empty caches, traffic spikes that blow past the cluster's total cache capacity. In those cases llm-d needs to fall back to pure load-based routing without adding cache-lookup overhead on top.

The EPP has a fast-path check for this: if no pod reports a cache entry matching the request's prefix hash, it skips the cache-aware routing logic entirely and routes on load metrics alone. No wasted time hunting for a cache match that doesn't exist — "all-cold" routing performs like a standard load balancer. As pods warm up and cache entries accumulate, the EPP shifts back to cache-aware routing on its own, no manual intervention or config change needed.

Same story at the prefix level. Deploy a new agent type with a system prompt no pod has cached, and llm-d treats its first request as a load-balanced cold start, building cache affinity from there. Within seconds the new prefix is cached and subsequent requests get cache-aware routing. It's automatic and invisible to the agent application — this "warm on first use" behavior means new agent types and system prompt updates need zero cache pre-warming. llm-d just handles it as part of normal request processing.

Conclusion: The Infrastructure Agentic AI Demands

Agentic workloads aren't some future thing — they're here now. Every major AI platform is shipping agent capabilities, and inference traffic is shifting from single-shot completions to multi-step, tool-augmented reasoning chains. That shift exposes a real inefficiency in traditional inference routing: recomputing the same shared prefixes over and over across an agent's call chain.

llm-d fixes this at the routing layer, which is where it belongs. By tracking which pods have cached which prefixes, and routing to maximize cache reuse while respecting load constraints, llm-d cuts up to 75% of prefill latency in agentic workloads. The savings compound across multi-turn conversations, multiply across multi-agent platforms, and scale as standardized tool protocols like MCP keep spreading.

The numbers speak for themselves: for an agent making 8 inference calls per execution, llm-d takes total prefill from 3.75 seconds to around 1.1 seconds. At 1,000 concurrent agents, that's nearly 44 GPUs' worth of freed compute per minute. For a cluster of 8x A100 GPUs at $2.21/hr each, cache-aware routing saves about $4,200/month at moderate agent traffic. And as agents get more complex (more tools, longer conversations, deeper multi-step reasoning), the savings only grow.

Beyond raw performance, llm-d makes agent architectures viable that latency used to rule out. ReAct loops can run longer without wrecking the user experience. Tool-augmented generation actually works for interactive apps. Multi-agent orchestration can scale to dozens of concurrent workers without flattening the inference cluster. MCP's whole vision of composable, tool-rich agents only works operationally if the infrastructure can handle the call volume those agents generate — and now it can.

The observability side matters just as much. With OpenTelemetry integration, agentic workloads become visible in a way they've never been. Slow agent execution? The trace shows exactly why — cache miss, tool timeout, overloaded pod. That turns agent performance work from guesswork into actual engineering. You can set SLOs on end-to-end agent latency, track cache hit rates per agent type, and catch routing inefficiencies before users feel them.

Basically, the agentic era needs inference infrastructure that understands the structure of its workloads. Random routing treats every request as independent. Cache-aware routing recognizes that in agentic systems, requests are deeply correlated — shared prefixes, shared sessions, shared tool definitions. llm-d is built to exploit those correlations, and that's why agentic workloads are its killer app.

If you're building agents today without cache-aware routing, you're paying for the same prefill computation 7 times per user request. Your users wait 2 to 4 seconds longer than they need to. Your GPU spend runs 30% to 50% higher than it should. The gap between what agents demand and what traditional LLM serving provides is wide and getting wider. llm-d closes it.