Re-measuring continuous batching, chunked prefill, and prefix caching with repeated measurements

Status: Draft (2026-10-27), Measured. Measured on 2026-10-27 using a single 8GB laptop GPU (Ada generation), vLLM 0.31.0, and Qwen3-Reranker-0.6B. Since the principles of the same three features and the initial measurements of the 1.7B model are covered in "Dissecting the LLM KV Cache" Part 2 and "Preparing Toss ML/AI Infra" Part 14, this post only covers what those articles did not (repeated measurements, disabled baseline, turning off chunks, and cache_salt). Added 2026-10-27. The results of re-running the same test with vLLM 0.32.0 on the same day have been added to Section 6-1. The 0.31.0 figures remain unchanged. Writing Rule: Measured data is the default; unmeasured items are marked as "Unmeasured", and unverified items as "Unverified".

Table of Contents

• 1. Why Re-measure?
• 2. Environment and Method
• 3. Continuous Batching: Disabling It Drops Throughput to 1/10
• 4. Chunked Prefill: Decode Pause Time is Proportional to Chunk Size
• 5. Prefix Caching: 18.9ms Even with Repetition
• 6. cache_salt: Preventing Tenants from Sharing Caches with Identical Prefixes
• 6-1. Re-measurement on 2026-10-27: Are Results Identical in 0.32.0?
• 7. Key Takeaways for Operations
• 8. Unverified Items
• References

1. Why Re-measure?

The principles of PagedAttention, continuous batching, and prefix caching within the serving engine's KV cache, along with the initial values measured using a 1.7B model, were covered in Direct Measurement of vLLM with a Smaller Model. Those measurements had three gaps:

1. Each condition was measured only once, so the variance was unknown.
2. Continuous batching could not be disabled, making it impossible to see "how it performs without it".
3. Cases with chunking completely disabled and isolation when multiple users share the cache (cache_salt) were not examined.

This time, all conditions were repeated three to four times, and the first batch was separated as a warm-up and excluded from the median. A server with the features disabled was launched immediately afterward on the same day in the same pod.

2. Environment and Method

• GPU: A single 8GB laptop GPU (Ada generation), driver 595.99. Container memory limit 4GiB.
• vLLM 0.31.0 official container (PyTorch 2.13.0, CUDA 13.0), attention backend shown as FLASH_ATTN in logs.
• Model: Qwen3-Reranker-0.6B, bf16, weights 1.12GiB. With 28 layers, 8 KV heads, and a head dimension of 128, the KV for a single token is 2 × 28 × 8 × 128 × 2 bytes = 114,688 bytes. This matches the log entry "40,208 tokens for 4.3GiB KV".
• Common server options: --max-model-len 4096 --gpu-memory-utilization 0.80. Otherwise, default values are used. In this version, if GPU memory is less than 70GB, the server sets the max_num_batched_tokens default to 2,048, and max_num_seqs to 256 (verified in the source vllm/engine/arg_utils.py ).
• Client: Sent streaming requests via aiohttp within the same pod to measure Time to First Token (TTFT) and Inter-Token Latency (ITL). The input consisted of 30 randomly concatenated English words, and the output length was fixed to ignore_eos .
• Repetition: For load testing, 4 batches per concurrency level were sent (the first batch being a warm-up), and the median of the remaining 3 batches was used. For prefix testing, 1 cold request and 4 warm requests were sent for each of 5 new prefixes.

3. Continuous Batching: Disabling It Drops Throughput to 1/10

There is no option to directly disable continuous batching. Instead, setting --max-num-seqs 1 to process only one request per step turns it into a "server processing requests one by one". Requests with 256 input tokens and 128 output tokens were sent at 2c requests for a concurrency level c.

The throughputs of the three batches were within 0.3% of each other (concurrency 16 default: 1,681.3, 1,683.1, 1,685.4).

• A server processing only one request keeps its throughput constant even when concurrency increases. Since a decode step mostly involves reading all weights from memory, the step duration does not vary significantly whether processing one request or sixteen. On the default server, the token interval at concurrency 16 was 8.2ms, only 1.5 times longer than when alone (5.6ms), while the throughput increased by 9.6 times.
• The price is waiting. On a single-request server, the median first token latency for 16 concurrent requests was 11 seconds. This is because requests had to queue until preceding ones finished.
• At concurrency 64, ITL p99 spiked to 90.6ms. This happens because new requests' prefills interleave between decode steps, and the chunk size in the next section determines this tail.

4. chunked prefill: Decode Pause Time is Proportional to Chunk Size

While a single decode request (900 tokens) was running, an input of approximately 3,700 tokens was interleaved five times, and the maximum decode token interval within the interleaved sections was measured.

On this GPU, the prefill speed is around 174.7ms for 3,200 tokens, or roughly 18,000 tokens per second. Calculations showing that a single 2,048-token chunk takes about 111ms and the entire 3,700 tokens take about 200ms match the maximum ITL in the table. The five values cluster within around 1ms, which is not a coincidence.

• Smaller chunks protect decoding while slightly delaying prefill. A 512-token chunk increases the interleaved input's TTFT from 216ms to 232ms by 7%, while reducing the decode's maximum stall by two-thirds.
• Disabling chunking brings behavior closer to the V0 era. In this case, max_num_batched_tokens must be greater than or equal to max_model_len for the server to start (same as the documentation warning).
• Under a concurrency 16 load (256 input tokens), a 512-token chunk did not change throughput (1,682.7 vs 1,683.1 tok/s), the TTFT median dropped from 131.0ms to 86.3ms, but ITL p99 increased from 9.4ms to 22.9ms. Under loads with many short inputs, the chunk size is merely a knob for choosing "which tail to reduce" rather than a free improvement. The cause of this difference is unverified as schedule logs were not examined.

The shape matches previous measurements taken with the 1.7B model (512 tokens: 69ms, 2,048 tokens: 227ms), scaled down in magnitude according to the model's computational load.

5. Prefix Caching: 18.9ms Even with Repetition

We appended 32 different words per request to a common prefix of 3,000 words (about 3,200 tokens). Five batches were run by changing the prefix.

Warm requests only compute the last block and tail of the 3,200 tokens, resulting in a 9.2x speedup. The very first request on the server with prefix caching disabled took 236ms, which was 60ms slower than other cold requests; this represents the warm-up overhead after launching the server. This is why the first value must be separated in measurements.

Keep in mind that caching operates at the block level. Since vLLM hashes and stores only full blocks (default 16 tokens), if the common part cuts off at a block boundary, the final fragment is recomputed. Hashing defaults to sha256 as of 0.11 and can be changed to --prefix-caching-hash-algo .

6. cache_salt: Preventing Tenants from Sharing Caches with Identical Prefixes

Prefix caching has a security issue. By sending the same beginning part as a document sent by another user, if the first token is fast, one can infer that "someone has already viewed this document." This is a timing attack leaking information via elapsed time. vLLM allows attaching cache_salt to requests to mix that value into the first block's hash, ensuring caches are shared only among identical salts.

--enable-prompt-tokens-details Using

to display the number of cached tokens in the response, we sent the same 3,000-word prefix across five batches with varying salts.

• Requests with the same salt reused up to the block boundary within the 3,200 tokens (3,184 or 3,200 tokens, which are multiples of 16).
• When salts differed, even prefixes identical down to a single character failed to utilize the cache, and the execution time was identical to a cold request. Requests without a salt also failed to use caches created by requests with a salt.
• The first request of the very first batch took 228.5ms, and the subsequent request took 27.6ms, slower than other batches. This is the warm-up overhead for the first request after starting the server.

cached_tokens appears in responses only when the server is launched with --enable-prompt-tokens-details . If disabled, this field is empty, requiring cache hits to be inferred solely from timing. Overall server hits can be checked via /metrics 's vllm:prefix_cache_hits_total and vllm:prefix_cache_queries_total .

6-1. Re-measurement on 2026-10-27: Are Results Identical in 0.32.0?

Re-measured in the vLLM 0.32.0 official container using the same GPU, same reranker model, and same client ("2026-10-27 0.31.0 Measurement → 0.32.0 Re-measurement").

All values were identical within 1.5%. The major changes in the 0.32.0 release notes (such as expanding Model Runner V2, removing cascade attention, max_tokens not pre-stripping from the context) did not change the numbers under this small model's baseline load. The conversational Qwen3-0.6B (same architecture) was also identical at 174.1, 1,678.0, and 2,898.7 tok/s at concurrency 1, 16, and 64.

7. Operational Takeaways

• If your service has high concurrent requests, continuous batching accounts for the majority of throughput. max_num_seqs lowering it saves KV cache at the direct expense of this benefit.
• If the decode tail latency is important, max_num_batched_tokens is decreased, and if the TTFT for long inputs is important, it is increased. On this GPU, the time for a single chunk was approximately the number of chunk tokens divided by 18,000 per second. By measuring your GPU's prefill speed once, you can work backward from your latency target to choose the chunk size.
• When multiple customers share a single engine, a cache_salt is attached per customer. Reuse within the same customer remains intact, while timing leaks between customers are prevented.
• When measuring, exclude the first request after starting the server, run the same conditions multiple times, and record the variance together. Once again, a single initial request was 35% slower than the other values.

8. Unverified Items

• Values for large models, multi-GPU setups, and real conversation data (workloads with mixed prefix lengths and reuse intervals) were not measured.
• The cause of the increased ITL p99 for short input workloads at a 512-token chunk size is unverified as schedule logs were not reviewed.
• While the release note stating that the default max_num_batched_tokens on large GPUs increased from 8,192 to 16,384 in 0.28.0 was confirmed, behavior on GPUs of that size was not measured.

References

• Kwon et al., "Efficient Memory Management for Large Language Model Serving with PagedAttention," SOSP 2023. 2–4x throughput compared to FasterTransformer and Orca at the same latency (per abstract). https://arxiv.org/abs/2309.06180
• Yu et al., "Orca: A Distributed Serving System for Transformer-Based Generative Models," OSDI 2022. https://www.usenix.org/conference/osdi22/presentation/yu
• Agrawal et al., "Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve," arXiv 2403.02310. 2.6x serving capacity of a single A100 for Mistral-7B compared to vLLM under tail latency constraints (per abstract). https://arxiv.org/abs/2403.02310
• vLLM Documentation, Optimization and Tuning (Chunked Prefill), Automatic Prefix Caching Design (cache_salt): https://docs.vllm.ai/en/latest/configuration/optimization.html, https://docs.vllm.ai/en/latest/design/prefix_caching.html
• Previous measurement: KV cache inside the serving engine, remeasuring vLLM directly with a small model