Every request an inference engine takes is, from its point of view, the first. The prompt arrives whole. The engine tokenizes it, schedules it into a batch, reuses whatever prefix it has cached, streams tokens back, and drops the whole thing. It owes the request nothing afterward, and that forgetting is why the engine is fast: batching, prefix caching, and scheduling all get simpler when no request carries obligations from the one before it.
An agent wants the opposite. A coding session is one conversation spread across dozens of model calls, with tool calls in the middle and their results fed back in, and by the third turn most of what the model needs to know is something it said itself earlier. Some of that was said to a tool and never to the user. Some of it was reasoning the model addressed to no one. Somebody has to hold all of it, and until recently that somebody was the client, resending the whole transcript on every turn.
vLLM's Agentic API (opens in new tab) is a Rust gateway that takes the job over. Reading its design records, I kept finding the same decision underneath each of them: the engine owns the tokens, the gateway owns the history, and much of the codebase is what that seam forces.
Where the seam goes
Figure 01 · the seam
The engine owns the tokens. The gateway owns the history.
three turns,
one store
One half of the rule is obvious and the other is not. Tokenization and chat templates stay in the engine. Every model family has its own template quirks and input sanitization, and the project declined to reimplement any of it. Its earliest design record points to llm-d (opens in new tab), a neighboring project that tried building its own tokenization, found it painful, and is converging back on vLLM's. So the gateway calls vLLM's own stateless Responses endpoint, sends the full history every time, and lets the engine do what it already does.
Less obviously, the history belongs to the gateway and to nobody else. A turn arrives with a previous_response_id and only the new input. From that id the gateway loads a checkpoint, appends the input, and hands the engine a complete prompt. Neither the engine nor the client has to remember the conversation: the engine never learns that there was a previous turn, and the client never has to prove it remembers one. State that used to be duplicated in every client now lives once, next to the engine, in a database.
The first consequence is how the history is stored. Each item is stored once, whether a user message, a model message, a tool call, a tool result, or a reasoning item. Every response keeps an ordered list of item ids, and that list is its continuation checkpoint. Embedding the whole history in every response row would make storage grow with the square of the conversation length, since each turn repeats every earlier one, and reasoning and tool payloads run far larger than the final text. Walking a chain of responses at request time would store less and read slower. The middle option reads one checkpoint, fetches its items in bulk, and restores their order in memory.
Clients that want none of this still get none of it. A request with no history, no storage, no compaction, and no gateway-owned tools goes through a byte-transparent proxy path to the engine, untouched. Statelessness stays available; it is no longer compulsory.
One turn, several rounds
Figure 02 · one turn
The client asks once. The engine may be asked several times.
up to ten rounds
behind one response
Because the gateway holds the history, the unit the client sees and the unit the engine sees can be different sizes. One request from the client asks for one response. Producing it can take several inference rounds, because the model may call a tool, and the tool's result has to go back into the next round before the answer can be finished. So the gateway runs that loop, at most ten rounds per turn, and projects the whole of it as one response and one continuous event stream, persisted once at the end.
Which tool runs where is decided by ownership, declared per tool before inference. Web search and MCP servers are gateway-owned: the gateway executes them mid-turn with bounded fan-out and feeds the results back. Plain functions, custom tools, and the namespaced tool groups that Codex declares are client-owned: their calls are returned unresolved for the client to answer, and the turn ends there. None of this distinction reaches the engine. Every tool it sees is a plain function declaration, normalized before the prompt is built, and every call comes back as a function call that the gateway routes by looking its name up in a registry built at the start of the request. Mixed rounds are allowed, too. Whatever the gateway owns runs, whatever the client owns comes back, and the response says which is which.
Routing by registry is a small choice and a load-bearing one: adding a new kind of tool means adding a handler on the gateway side, while the engine keeps a single, simple notion of a tool.
What remembering costs
Any checkpoint that grows forever will eventually exceed the context window, so the gateway compacts. It asks the model to summarize the history, stores the summary as a new item, and issues a new checkpoint that carries the user's own messages forward and ends with the summary. Old rows are left alone, and every earlier response id still works as a starting point. If the summary turns out to be poor, or a client wants the uncompacted branch, the earlier checkpoint is still there to continue from.
Deciding when to compact runs into the seam directly. The gateway has no tokenizer of its own, so it estimates instead, the same way for every model: bytes of text divided by four, a fixed allowance for each item's framing, a flat 1,024 tokens per image regardless of the image. Its documentation says the estimate is not a model-specific tokenizer and tells you to leave headroom. Owning the history means owning its size, and not owning the tokens means measuring that size with a ruler that is approximate on purpose.
Reasoning items are the part I keep returning to. With current models, some of what the next turn depends on is the model's own earlier reasoning. Whatever copy of that reasoning the client keeps, the one replayed into that turn's prompt comes from the gateway's store, and the conversation, in the sense that matters to the model, is no longer something the client has to hold. A design that lets the engine forget has to put it there, and that is a strange place for a transcript to end up.
