The 25% Page-Size Slowdown That Wasn't

The result looked almost too clean: set SGLang's KV-cache page size to 1 and time per-output-token rose by 11–25% against page size 128. The gap widened with batch size. It survived fresh processes, reversed run order, repeated trials, and an exclusive GPU. At the closest reconstruction of the original configuration, the difference was larger still: 67.41 ms versus 51.27 ms, or 31.5%.

This was exactly the kind of result that invites a plausible story. Smaller pages imply more page entries; more entries imply more index work; more scattered addresses imply worse memory behavior. The explanation practically wrote itself.

It was also wrong.

The steady decode steps from those same runs were 43.80 ms and 43.83 ms. Across 475 full-batch steps, page size changed decode latency by −0.07% — effectively nothing. The 31.5% existed in the benchmark's output, but not in the operation that output claimed to measure.

That distinction became the most useful lesson in the investigation. A systems result can be reproducible, statistically tidy, and mechanistically false at the same time. Repetition tells you that a number is stable. It does not tell you what the number means.

Four meanings of “page size”

A KV cache stores the keys and values produced by previous tokens so autoregressive decoding does not recompute them. Paged KV-cache systems add a level of indirection: each sequence owns logical pages that map onto physical cache storage. This makes allocation, sharing, and eviction manageable under continuous batching.

But page_size is not one mechanism. It can enter the system at least four different layers:

1. Allocation granularity. How many token slots are reserved and freed together.
2. Scheduler and prefix-cache bookkeeping. How cached prefixes are matched, attached, and represented.
3. Backend lowering. Whether the serving engine passes per-page metadata or expands it into per-token indices.
4. Kernel execution. Whether the attention kernel's instruction count, memory transactions, or launch plan depends on the number of pages.

Conflating these layers is dangerous. A page-size flag visible at the command line may disappear before the kernel. Conversely, a kernel that consumes pages natively may make the same flag a real device-side parameter. Before predicting performance, I now ask a more basic question:

Which representation reaches the timed operation?

For the studied SGLang/FlashInfer path, the answer was surprising. The KV pool was token-flat, normal decode built a kv_indices list of length B × context_length at every engine page size, and the FlashInfer plan received device page size 1. Engine page_size=1 and page_size=128 therefore did not create a 128× index-length contrast in the decode path. They changed allocator behavior and potentially the values — the physical token locations — inside an equally long index list.

That source trace killed the first explanation before profiling did. The hypothesized O(number_of_pages) decode work did not exist in this build.

The metric crossed a phase boundary

The benchmark defined time per output token as:

TPOT_harness = (batch_end - first_token_time) / (output_tokens - 1)

For a single request that formula is fine. This was not a single request. first_token_time was the first streamed chunk from any request, while batch_end was the completion of all of them, and the harness flushed the radix cache before every repeat and submitted a batch of identical long prompts, so the two boundary events ended up on opposite sides of a phase change.

Those details create a mixed phase:

first request emits
        │
        ├── other requests still prefill / attach to the cold prefix
        ├── batch membership ramps toward B
        └── steady full-batch decode begins
                                            │
                                            └── all requests finish

The interval labeled “decode” was carrying something else. It contained a cold shared-prefix admission ramp: requests completed chunked prefill at different times, and the earliest one started generating while the rest were still joining the batch. At batch 32 that ramp ran about 2.2 seconds at page size 1 against 0.7 seconds at page size 128. Divide that difference by 95 output-token intervals and you have manufactured 16 milliseconds per token out of nothing.

A more honest decomposition is:

T_reported(p) ≈ T_steady_decode(p)
              + T_cold_admission(p) / (N_output - 1)

The experiment had discovered a real page-sensitive cost, but assigned it to the wrong phase. The excess time lived in cold shared-prefix admission — not in per-token decode.

Three checks made this diagnosis decisive.

First, the engine's own full-batch step logs were flat: 43.80 versus 43.83 ms. Second, changing only the generated length from 96 to 512 tokens collapsed the reported gap from 31.7% to 6.3%, while steady decode remained unchanged. A fixed ramp divided by more tokens must amortize; a recurring kernel penalty should not. Third, a longer 16.4k-token prompt made the reported gap disappear even though attention occupied at least as much of the decode step. The effect followed admission geometry, not attention work.

When source beats intuition

After locating the timing problem, I still wanted to bound any smaller page effect hidden underneath it. There were two candidate paths: host-side metadata preparation and device-side gathers.

The host hypothesis predicted that tiny pages would produce much longer index lists and more planning work. Source inspection showed that both page sizes produced the same B × L token indices. A synthetic timing sweep then measured the FlashInfer plan() call at roughly 0.26 ms from 64 through 2,097,152 entries. Even the artificial length contrast the engine did not create was flat at this scale.

The device hypothesis predicted that page size 1 could fragment physical token locations and turn contiguous reads into scattered gathers. To isolate the worst case, I held the token-flat KV tensor and index length constant, then compared three index layouts:

• contiguous: sequential token locations;
• block128: contiguous 128-token runs in shuffled block order;
• scatter: a random permutation, deliberately harsher than a fresh allocator.

Across footprints from 0.03 to 8.59 GB, maximal scatter changed FlashInfer decode by −1.0% to +1.3%. At the largest one-million-index cell, contiguous measured 10.1418 ms and scatter 10.2409 ms: a 0.98% difference.

Nsight Compute explained why this backend was insensitive in the measured distinct-KV regime. Contiguous and scattered runs both sustained about 96% of peak DRAM throughput, both had 0.7% L2 hit rate, and both issued the same 15.6 sectors per request. Each token's KV payload was a contiguous 4 KB burst; shuffling which 4 KB region came next did not change the efficiency of the burst itself. The kernel was already constrained by streaming bytes from DRAM, so the small address-order difference did not move the wall.

This is an important non-result. “Scattered” is not automatically synonymous with “slow.” The relevant question is which transaction pattern changes, and whether that change is exposed by the current bottleneck.

A backend where pages really do reach the kernel

The cleanest way to test the resulting theory was to find a positive control: a backend whose device work genuinely depends on page count.

On the same sm120 system, SGLang's trtllm_mha decode path selected the XQA kernel. The split engine configuration used XQA for decode and FlashInfer for prefill; the genuine trtllm-gen FMHA path was unsupported on sm120. Unlike the token-flat FlashInfer path above, XQA accepted page-aware metadata. Nsight Compute showed a strong, deterministic page-count component in global-load instructions.

The engine accepted page sizes 16, 32, and 64; page size 1 was inexpressible and 128 was coerced to 64. Direct XQA kernel probes could exercise 128, which is why the distinct-KV counter control below compares page 16 with page 128 while the engine-facing table stops at 64.

For a shared-prefix batch at KV length 8,192:

The page-16 kernel executed 2.38× as many global-load instructions as page 64 and ran 9.6% longer. Reprofiling reproduced the load count exactly and DRAM-read bytes within 0.2%, which matters because profiler replay can otherwise create seductive counter artifacts.

This was the real page-size effect the original story had imagined — just in a different backend and for a measured reason. Smaller pages created more device work, and the work extended kernel duration.

Yet even here, page sensitivity was conditional.

With distinct KV streams, the kernel ran at roughly 95% of peak DRAM throughput. Page 16 still executed more global loads than larger pages, but kernel time stayed at about 327 µs. Changing the page overhead did not remove the compulsory KV stream, and the measured kernel remained near the DRAM wall at every page size. The extra work existed without being performance-visible.

With a shared prefix, the picture inverted. Requests reused the same KV data within the kernel, L2 hit rate rose to about 95–97%, and DRAM throughput fell far below saturation. The compulsory byte stream was no longer the dominant term. Page-count-dependent issue work could now sit on the critical path, so larger pages became measurably faster.

Shared prefixes did not help because the footprint was merely “small.” They helped because multiple requests reused the same cache lines during the kernel. Residency and reuse are not interchangeable: a compact set of unrelated streams can still miss, while a larger shared prefix can hit repeatedly.

A critical-path model, not a universal law

The measurements suggest a compact diagnostic model:

T_step(p) ≈ T_engine + max(
    T_DRAM,
    T_issue(p),
    T_math
)

The equation predicts nothing. It tells you which of the three terms to go and measure.

Page size can affect latency only if two conditions hold:

d(device_or_host_work) / d(page_size) ≠ 0

and

that work contributes to the active critical path

The first condition is about representation. FlashInfer's studied SGLang lowering erased page-count-dependent decode work by expanding to a fixed-length per-token representation. XQA preserved page-aware work, so its global-load count changed with page size.

The second condition is about regime. In an off-wall, high-reuse XQA cell, reducing page work lowered time. In a DRAM-saturated distinct-KV cell, it did not. A counter can move by 2× while latency remains flat if another term dominates the maximum.

This also explains why isolated percentage claims travel poorly. “Page size 64 is 10% faster” is not a property of 64. It is a property of a backend, lowering, GPU, batch shape, context length, cache-reuse pattern, and measurement boundary. Change any of those and the causal path may disappear.

The workflow I wish I had started with

The investigation eventually converged on a sequence that is faster than inventing a mechanism from a benchmark table:

1. Draw the timing boundary. Mark prefill, admission, steady decode, draining, and synchronization. Identify whether the start and end events refer to one request or the whole batch.
2. Log the operation directly. For decode claims, collect per-step latency only after the batch reaches a stable size. Keep TTFT, inter-token latency, and whole-call throughput separate.
3. Trace the flag to the kernel. Record tensor shapes, index lengths, page-table lengths, backend selection, and any internal page-size coercion. Do not infer device behavior from a CLI name.
4. Construct positive and negative controls. Compare shared versus distinct KV, contiguous versus adversarial scatter, and on-wall versus off-wall cells. A causal explanation should predict where the effect vanishes.
5. Profile the smallest decisive cell. Use CUDA events for duration and Nsight Compute for instruction count, DRAM bytes, cache hit rate, and sectors per request. Reprofile headline cells to test replay stability.
6. Attack the proposed mechanism. Change output length to test amortization; change context length to test scaling; switch backends to test whether the representation survives lowering.

The order matters. Profiling a mislabeled interval in exquisite detail only produces a more expensive misunderstanding.

Scope and evidence boundaries

These results come from several deliberately separate arms on one exclusive RTX 5060 Ti (sm120, 16 GB), not from one monolithic benchmark:

• Whole-call reconciliation. Qwen3-VL-2B, shared prefix ≈9.7k tokens, batch 32, Triton, radix cache and CUDA graph enabled. The behaviorally matched SGLang rebuild used commit 0eded9e208, PyTorch 2.9.1+cu128, sgl-kernel 0.3.21, FlashInfer 0.6.6, and num_kv_splits=8. The historical checkout itself was not recovered; its original logs independently contain the same page-flat steady-step result.
• FlashInfer host and gather isolation. An engine-faithful, weightless harness preserved the token-flat KV layout and per-token indices while sweeping synthetic plan lengths and index contiguity. This arm bounds proposed host and gather costs; it is not an end-to-end model-serving measurement.
• XQA positive control. The shared-profile cell used batch 32 and KV length 8,192. The distinct-profile counter cell used batch 8 and KV length 4,096. They test the same page-aware kernel under different reuse and bandwidth regimes; they are not workload-matched latency competitors.

The values labeled as timings, instruction counts, bandwidth, and cache hit rates are measured. The critical-path equation is a diagnostic interpretation, not a fitted performance model. The cold-ramp evidence localizes the old whole-call gap to shared-prefix admission, but it does not time-separate radix matching, allocation, and scheduler bookkeeping inside that interval. The headline XQA page-16/page-64 counters were independently replayed; not every point in the wider context-length sweep was.

What the result says

On the measured SGLang/FlashInfer build, normal decode used token-flat indices and was page-invariant within the tested range; worst-case synthetic scatter stayed within 1.3%; the old double-digit shared-prefix result was a cold-admission term folded into a TPOT formula. On the measured XQA path, page size really did change global-load work, and larger supported pages improved shared-prefix off-wall kernel time by roughly 10% while distinct, DRAM-saturated cells stayed flat. Both halves are load-bearing. One is a counterexample to any claim that page size never matters; the other is a warning against claiming that it always does.

This is a single-GPU, single-stack study. It establishes counterexamples and a method, not a portability claim for other architectures, backend versions, models, or attention layouts. The broader finding is methodological:

A configuration parameter matters only when it changes the work that reaches the measured operation — and that work reaches the critical path.

The first benchmark gave me a compelling percentage. The source trace told me the proposed mechanism was absent. The step logs exposed the wrong timing boundary. The profiler found a different backend where the mechanism was real. That progression — from number, to boundary, to representation, to bottleneck — is the part I expect to reuse. I already have, on a measurement problem with no GPU in it at all.

The specific percentages will age with kernels and hardware. The questions should age better.