The 31% Page-Size Slowdown That Wasn't

The result looked almost too clean: with SGLang's KV-cache page size set to 1, time per output token (TPOT) rose 11–25% against page size 128. The gap widened with batch size. It survived fresh processes, reversed run order, repeated trials, and an exclusive GPU. At the closest reconstruction of the original configuration, the difference was larger still: 67.41 ms versus 51.27 ms, or 31.5%.

This was the kind of result that invites a plausible story. Smaller pages imply more page entries; more entries imply more index work; more scattered addresses imply worse memory behavior. The explanation practically wrote itself. I believed it long enough to write the gap up as a decode-kernel penalty.

It was also wrong.

The steady decode steps from those same runs were 43.80 ms and 43.83 ms. Across 475 full-batch steps, page size changed decode latency by −0.07% — effectively nothing. The 31.5% existed in the benchmark's output, but not in the operation that output claimed to measure.

A systems result can be reproducible, statistically tidy, and mechanistically false at the same time. Repetition tells you that a number is stable. It does not tell you what the number means.

Four meanings of “page size”

A KV cache stores the keys and values produced by previous tokens so autoregressive decoding does not recompute them. Paged KV-cache systems add a level of indirection: each sequence owns logical pages that map onto physical cache storage. This makes allocation, sharing, and eviction manageable under continuous batching.

But page_size is not one mechanism. It can enter the system at any of at least four layers:

1. Allocation granularity. How many token slots are reserved and freed together.
2. Scheduler and prefix-cache bookkeeping. How cached prefixes are matched, attached, and represented.
3. Backend lowering. Whether the serving engine passes per-page metadata or expands it into per-token indices.
4. Kernel execution. Whether the attention kernel's instruction count, memory transactions, or launch plan depends on the number of pages.

A page-size flag visible at the command line may disappear before the kernel. Conversely, a kernel that consumes pages natively may make the same flag a real device-side parameter. Before predicting performance, I now ask a more basic question:

Which representation reaches the timed operation?

For the SGLang/FlashInfer path I studied, the answer contradicted the plausible story. The KV pool was token-flat, normal decode built a kv_indices list of length B × context_length at every engine page size, and the FlashInfer plan received device page size 1. Engine page_size=1 and page_size=128 therefore did not create a 128× index-length contrast in the decode path. They changed allocator behavior and potentially the values — the physical token locations — inside an equally long index list.

That source trace killed the first explanation before profiling did. The hypothesized O(number_of_pages) decode work did not exist in this build. Tracing the Triton backend that produced the 31.5% turned up the same per-token list and a decode kernel that took no page-size argument.

The metric crossed a phase boundary

The benchmark defined time per output token as:

TPOT_harness = (batch_end - first_token_time) / (output_tokens - 1)

For a single request that formula is fine. This was not a single request. first_token_time was the first streamed chunk from any request, while batch_end was the completion of all of them, and the harness flushed the radix cache (SGLang's shared-prefix cache) before every repeat and submitted a batch of identical long prompts, so the two boundary events ended up on opposite sides of a phase change.

The interval labeled “decode” therefore contained a cold shared-prefix admission ramp, the mixed phase Figure 01 draws: requests completed chunked prefill at different times, and the earliest one started generating while the rest were still joining the batch. At batch 32 that ramp ran about 2.2 seconds at page size 1 against 0.7 seconds at page size 128. Spread that 1.5-second difference over the 95 intervals between 96 output tokens and you have charged roughly 16 ms per token to decode steps that never spent it. That is the whole gap between 67.41 and 51.27 ms.

Split into its two phases, the reported number is:

T_reported(p) ≈ T_steady_decode(p)
              + T_cold_admission(p) / (N_output - 1)

The benchmark had found a real page-sensitive cost but assigned it to the wrong phase. Three checks made this diagnosis decisive.

First, the engine's own full-batch step logs were flat: 43.80 versus 43.83 ms. Second, a fixed ramp divided by more tokens must amortize; a recurring kernel penalty should not. A separate comparison on that configuration changed only the generated length, from 96 to 512 tokens, and the reported gap collapsed from 31.7% to 6.3% while steady decode remained unchanged. Multiplied by their 95 and 511 intervals, the per-token differences behind those two percentages each come to about 1.5 seconds. Third, a longer 16.4k-token prompt made the reported gap disappear even though attention occupied at least as much of the decode step. At that length the rest of the batch joined in the prefill pass that produced the first token or in the next one, at either page size, so little of the admission fell inside the timed interval. The effect followed admission geometry, not attention work.

When source beats intuition

The timing artifact did not rule out a smaller page effect hidden underneath it. While still hunting for a mechanism, I had already measured the two candidate paths: host-side metadata preparation and device-side gathers.

The host hypothesis predicted that tiny pages would produce much longer index lists and more planning work. Source inspection showed that both page sizes produced the same B × L token indices. A synthetic timing sweep then measured the FlashInfer plan() call at roughly 0.26 ms from 64 through 2,097,152 entries. Even the artificial length contrast the engine did not create was flat at this scale.

The device hypothesis predicted that page size 1 could fragment physical token locations and turn contiguous reads into scattered gathers. To isolate the worst case, I held the token-flat KV tensor and index length constant, then compared three index layouts:

• contiguous: sequential token locations;
• block128: contiguous 128-token runs in shuffled block order;
• scatter: a random permutation, deliberately harsher than a fresh allocator.

Across footprints from 0.03 to 8.59 GB, maximal scatter changed FlashInfer decode by −1.0% to +1.3%. At the largest one-million-index cell, contiguous measured 10.1418 ms and scatter 10.2409 ms: a 0.98% difference.

Nsight Compute explained why this backend was insensitive in the measured distinct-KV regime. Contiguous and scattered runs both sustained about 96% of peak DRAM throughput, both had a 0.7% L2 hit rate, and both issued the same 15.6 sectors per request. Each token's KV payload was a contiguous 4 KB burst; shuffling which 4 KB region came next did not change the efficiency of the burst itself. The kernel was already constrained by streaming bytes from DRAM, so the small address-order difference did not move the wall. Whether scattered addresses slow a kernel depends on which transaction pattern changes and whether the current bottleneck exposes that change.

A backend where pages really do reach the kernel

The cleanest way to test the resulting theory was to find a positive control: a backend whose device work genuinely depends on page count.

On the same sm120 system, SGLang's trtllm_mha decode path selected the XQA kernel, TensorRT-LLM's decode attention kernel, which takes page-aware metadata. The split engine configuration used XQA for decode and FlashInfer for prefill; the genuine trtllm-gen FMHA path was unsupported on sm120. Unlike the token-flat FlashInfer path above, XQA kept the pages. Nsight Compute showed a strong, deterministic page-count component in global-load instructions.

The engine accepted page sizes 16, 32, and 64; page size 1 was inexpressible and 128 was coerced to 64. Direct XQA kernel probes could exercise 128, which is why the distinct-KV counter control below compares page 16 with page 128 while the engine-facing table stops at 64.

For a shared-prefix batch at KV length 8,192:

The page-16 kernel executed 2.38× as many global-load instructions as page 64 and ran 9.6% longer. Reprofiling reproduced the load count exactly and DRAM-read bytes within 0.2%, which matters because profiler replay can otherwise create seductive counter artifacts.

This was the real page-size effect I had imagined at the start — just in a different backend and for a measured reason. Smaller pages created more device work, and the work extended kernel duration.

Yet even here, page sensitivity was conditional. With distinct KV streams the kernel ran at roughly 95% of peak DRAM throughput and its time stayed at about 327 µs across page sizes: page 16 still issued more global loads, but the compulsory KV stream kept the kernel at the DRAM wall, so the extra work existed without being performance-visible.

With a shared prefix, the picture inverted. Requests reused the same KV data within the kernel, L2 hit rate rose to about 95–97%, and DRAM throughput fell far below saturation. The compulsory byte stream was no longer the dominant term. Page-count-dependent issue work could now sit on the critical path, so larger pages became measurably faster.

Shared prefixes did not help because the footprint was merely “small.” They helped because multiple requests reused the same cache lines during the kernel. Residency and reuse are not interchangeable: a compact set of unrelated streams can still miss, while a larger shared prefix can hit repeatedly.

A critical-path model, not a universal law

The measurements suggest a compact diagnostic model:

T_step(p) ≈ T_engine + max(
    T_DRAM,
    T_issue(p),
    T_math
)

Nothing in the equation is fitted to data. It tells you which of the three terms to go and measure.

Page size can affect latency only if two conditions hold:

d(device_or_host_work) / d(page_size) ≠ 0

and

that work contributes to the active critical path

The first condition is about representation. In the SGLang build I studied, lowering to FlashInfer erased page-count-dependent decode work by expanding to a fixed-length per-token representation. XQA preserved page-aware work, so its global-load count changed with page size.

The second condition is about regime. In an off-wall, high-reuse XQA cell, reducing page work lowered time. In a DRAM-saturated distinct-KV cell, it did not. A counter can move by 2× while latency remains flat if another term dominates the maximum.

This also explains why isolated percentage claims travel poorly. “Page size 64 is 10% faster” is not a property of 64. It is a property of a backend, lowering, GPU, batch shape, context length, cache-reuse pattern, and measurement boundary. Change any of those and the causal path may disappear.

The workflow I wish I had started with

I eventually converged on a sequence that is faster than inventing a mechanism from a benchmark table:

1. Draw the timing boundary. Mark prefill, admission, steady decode, draining, and synchronization. Identify whether the start and end events refer to one request or the whole batch.
2. Log the operation directly. For decode claims, collect per-step latency only after the batch reaches a stable size. Keep TTFT, inter-token latency, and whole-call throughput separate.
3. Trace the flag to the kernel. Record tensor shapes, index lengths, page-table lengths, backend selection, and any internal page-size coercion. Do not infer device behavior from a CLI name.
4. Construct positive and negative controls. Compare shared versus distinct KV, contiguous versus adversarial scatter, and on-wall versus off-wall cells. A causal explanation should predict where the effect vanishes.
5. Profile the smallest decisive cell. Use CUDA events for duration and Nsight Compute for instruction count, DRAM bytes, cache hit rate, and sectors per request. Reprofile headline cells to test replay stability.
6. Attack the proposed mechanism. Change output length to test amortization; change context length to test scaling; switch backends to test whether the representation survives lowering.

The order matters, and I applied steps 1 and 2 to the shared-prefix cell only four weeks after the original runs. Those June runs had per-step decode logging switched on, and I read their log lines to confirm that every request fit in the batch. The per-step throughput printed on those lines went unread while I profiled kernels. Step 2 would have settled the question in June. Profiling a mislabeled interval in exquisite detail only produces a more expensive misunderstanding. I have since used this order on a measurement problem with no GPU in it at all.

Scope and evidence boundaries

These results come from several deliberately separate arms on one exclusive RTX 5060 Ti (sm120, 16 GB), not from one monolithic benchmark:

• Whole-call reconciliation. Qwen3-VL-2B, shared prefix ≈9.7k tokens, batch 32, Triton, radix cache and CUDA graph enabled. The behaviorally matched SGLang rebuild used commit 0eded9e208, PyTorch 2.9.1+cu128, sgl-kernel 0.3.21, FlashInfer 0.6.6, and num_kv_splits=8. The historical checkout itself was not recovered; its original logs independently contain the same page-flat steady-step result.
• FlashInfer host and gather isolation. An engine-faithful, weightless harness preserved the token-flat KV layout and per-token indices while sweeping synthetic plan lengths and index contiguity. This arm bounds proposed host and gather costs; it is not an end-to-end model-serving measurement.
• XQA positive control. The shared-profile cell used batch 32 and KV length 8,192. The distinct-profile counter cell used batch 8 and KV length 4,096. They test the same page-aware kernel under different reuse and bandwidth regimes; they are not workload-matched latency competitors.

The values labeled as timings, instruction counts, bandwidth, and cache hit rates are measured. The critical-path equation is a diagnostic interpretation, not a fitted performance model. The cold-ramp evidence localizes the old whole-call gap to shared-prefix admission, but it does not time-separate radix matching, allocation, and scheduler bookkeeping inside that interval. The headline XQA page-16/page-64 counters were independently replayed; not every point in the wider context-length sweep was. The sixteen reports behind this section, with their measurement logs, profiler exports, and the scripts that turned them into figures, are kept together with their provenance (opens in new tab), so any number here can be followed back to the run that produced it.

What the result says

On the measured SGLang/FlashInfer build, normal decode used token-flat indices and was page-invariant within the tested range; worst-case synthetic scatter stayed within 1.3%; the old double-digit shared-prefix result, measured on Triton, was a cold-admission term folded into a TPOT formula. On the measured XQA path, page size really did change global-load work, and larger supported pages improved shared-prefix off-wall kernel time by roughly 10% while distinct, DRAM-saturated cells stayed flat. The XQA half is a counterexample to any claim that page size never matters; the FlashInfer half is a warning against claiming that it always does.

This is a single-GPU, single-stack study. It establishes counterexamples and a method, not a portability claim for other architectures, backend versions, models, or attention layouts.

The first benchmark gave me a compelling percentage. The source trace told me the proposed mechanism was absent. The step logs exposed the wrong timing boundary. The profiler found a different backend where the mechanism was real.