The Agent Era Broke Inference Serving. Here’s What to Fix.

I have spent the past year on two things: making inference frameworks serve Agent traffic well, and letting Agents do Infra work. They sound like two sides of one problem. In practice they have different bottlenecks, different metrics, and different engineering constraints. When I say “AI Infra for the Agent era,” I mean two things. First: how does an inference framework run Agent requests fast and reliably? Second: how do we get Agents to build and optimize Infra? The first is a systems problem. The second is a workflow problem.

0x1 Serving Agent traffic: from stateless requests to stateful trajectories

Agent traffic does not look like chatbot traffic. A coding task often runs dozens or hundreds of turns. Each turn appends a bit of tool output to a long context, then asks the model to keep writing. The prefix is almost entirely repeated. Last week DeepInfer’s technical report gave a number: the online prefill KV cache hit rate is 96%. Only about 4% of tokens need to be computed. The core of the serving stack becomes KV cache management.

One misreading to avoid: 96% hit rate does not mean only 4% of the work remains. It means prefill prefix reuse is high, and the bottleneck moves from “computing prefill” to “finding cache, routing to cache, moving cache, managing cache.” Decode still runs the full context every step. Long-context attention and bandwidth pressure do not disappear. Worse, when the hit rate is high, cache lookup, global indexing, tenant isolation, eviction policy, and KV transfer between prefill and decode after PD separation become new sources of tail latency. The KV cache system, PD separation, and load balancing are the central components for Agent traffic. When DeepInfer open-sourced last week, those pieces were not fully released. That makes benchmark reproducibility weaker: you may be comparing SGLang plus an unreleased surrounding system, not SGLang alone.

Storage tiering is inevitable. When HBM runs out, KV cache goes to CPU memory, SSD, or remote storage. SGLang’s HiCache and DeepInfer’s 3FS do this. A request arrives and must be routed by prefix to the machine holding that cache segment. After PD separation, KV moves between prefill and decode. These are scheduling problems: matching a request to the machine that holds its cache without blowing up the hot cache. Prefix routing cannot rely on prefix hash alone. It has to handle multi-replica cache consistency, tenant isolation, capacity tiering, fallback after eviction, and PD transfer topology. Once long-context requests dominate, bugs that short-request stress tests never exposed start showing up. There are many of them, and they need fixing. SGLang has fixed many of these.

Then speed. An Agent task runs many turns. Decode speed per turn determines how long the whole task takes. So the field has started competing on per-user tokens/s. I agree with the direction, but I complained about this under the MegaKernel question: do not fixate on bs=1. A production Agent service runs tens or hundreds of concurrent requests. At bs=64/128, you need single-user speed and aggregate throughput together. That comes from speculative decoding (DSpark, MTP), PDL, kernel fusion, and communication overlap. Speculative decoding yields less at large batch sizes because verification overhead and draft length need dynamic tuning. PDL, kernel fusion, and comm overlap work, but they are complex to engineer. The metric should be: at bs=64/128, under a given TTFT/TPOT SLO, look at per-user speed, aggregate throughput, and tail latency together. Otherwise you optimize a framework that is fast per request and miserable in the queue.

Agentic RL adds another set of constraints. RL rollout is inference, with a more complicated flow: multi-turn, long tail, tool calls that block mid-trajectory, and a training-side requirement that rollout logprobs match the training side. The inference framework must abort at any time, support partial rollout, hot-update weights, and keep kernels batch invariant so results reproduce. SGLang started working on deterministic inference last year because RL pushed it there. An RL rollout inference framework looks more like an interruptible, reproducible, hot-updatable batch system than an online serving system. The requirements overlap with online serving, but they are not the same.

So I would draw this layer as two layers: a session-state layer and a compute layer. The session-state layer handles KV cache, prefix routing, storage tiering, PD transfer, and load balancing. The compute layer handles decode speed, speculative decoding, kernel fusion, and communication overlap. The bottlenecks and optimization levers differ. Mixing them makes people stare only at kernels. The main risk in the Agent era is celebrating a faster kernel while cache routing blows up tail latency.

0x2 Letting the Agent do Infra: humans move from writing code to defining what counts as work

After GPT6 and Opus5.5, I basically stopped writing code by hand. I push Agents to do the work. Give a /goal that specifies the model, hardware, performance target, and acceptance criteria, and the Agent profiles, finds the bottleneck, writes the kernel, and retests. Human experience still helps. Some skills remain useful: aligning context, separating prefill and decode, mapping a kernel to the Python line that calls it, knowing under which dtype and shape a fusion holds, checking whether someone upstream already opened a similar PR. An Agent can figure these out, but it wastes time. So I organized them into profile-related skills and 118 model-PR history archives, so the Agent can look them up cold. The repo is at https://github.com/BBuf/AI-Infra-Auto-Driven-SKILLS/tree/main/skills. This is feeding the Agent context priors instead of making it rediscover them.

Reward hacking is the hard part to describe. Anyone who has run Agents has been burned by it. The Agent makes the metric look good while bypassing the bottleneck. It edits the test until it passes while breaking an invariant. It produces a kernel that is fast for one shape and falls over on another dtype. Prompting does not fix reward hacking. Verifiable rewards, sandboxes, evaluation isolation, cross-agent validation, and performance gates do. Human value shifts from writing lines of code to defining what counts as work and how to prove it works.

SGLang now gets dozens to over a hundred PRs a day, most submitted directly by Agents. Human review cannot read them all. What I care about is whether Agent code invasively breaks the runtime. sglang.kernels has been refactored several times. Kernels are grouped by function, classification sanity tests were added, and Agent-generated kernels such as KDA and Cake are placed in separate backends with their source labeled at registration. Nobody reads those line by line. The Agent iterates until correctness holds. Human review moves from syntax to guardrails: are invariants broken, is there a performance regression gate, are kernel sources and categories clear, are interfaces stable. Humans review interfaces and guardrails; Agents iterate implementation and correctness.

0x3 Open source has no moat, but the last mile does

I keep coming back to this: open-source inference frameworks, or open-source software generally, have no moat. Last week SGLang compared itself against DeepInfer on DeepInfer’s own benchmark. DeepSeek’s kernels, DSpark, and SWA bounded replay are all in SGLang main. Prefill was at most 1.04x faster, and decode was at most 7.4% shorter per step. Part of that is because SGLang uses DeepInfer’s open-sourced operators. Once something is open-sourced, it gets absorbed.

So where does the difference live? Scenario data, evaluation systems, SLA operations, failure handling, trust, and ecosystem. The framework itself has no moat. Knowing what to optimize, how to prove it got faster, and how to keep it from falling over under production traffic does. Automation is the outcome, but the human interface for defining what counts as work does not disappear. It moves from writing code to writing goals, acceptance criteria, and guardrails.

The job in the Agent era is to pick the right bottleneck, design verifiable acceptance, and guard runtime invariants. Code details go to the Agent. Humans will not understand them, and that is normal. One addition: do not let automation become auto-breakage. Performance gates and regression tests have to move faster than the Agent writes code. Automation is the only outcome left. The guardrails need to arrive before it does.