Measuring vLLM directly with a small model: Load, KV cache capacity, prefix cache, chunk size

Status: Draft (2026-10-07), empirical measurement post. All numbers are directly measured by the author in the environment below; calculations are marked as 「Calculated」 and unmeasured items as 「Unmeasured」. Since this is a single environment, a single model, and a single batch of tests, please focus more on the shapes and principlesrather than absolute values. Writing rule: Empirical data is the baseline, and unmeasured items are marked as 「Unmeasured」.

Table of Contents

• 1. What I wanted to know
• 2. Environment and methodology
• 3. KV cache capacity matches the formula
• 4. Increasing concurrent requests: Throughput slows down, then the queue explodes at KV capacity limit
  • When KV is insufficient: Do they just wait, or are they evicted?
  • Max concurrent sequences (--max-num-seqs) only shifts the queue location
• 5. KV 8-bit: Increased capacity reduces long input latency
• 6. Prefix cache: Shared prefix makes the first token 14x faster
• 7. Chunked prefill: Chunk size determines how long a long input pauses decoding
• 8. Disabling CUDA graphs (--enforce-eager)
• 9. Initial server startup time
• 10. Takeaways for operations
• 11. Potential interview questions
• External basis: Verified with papers and official documentation
• Unverified items
• Verified versions and dates

1. What I wanted to know

The vLLM documentation has many options, and in interviews and operations, people ask 「What changes and by how much when you turn on this option?」 Part 4 summarized the principles (KV cache formula, prefix cache, scheduler) through documentation, and this post answers the same questions with numbers.

1. Where do throughput and latency start to degrade as concurrent requests increase?
2. Does the KV cache capacity match the calculated formula, and how does that capacity connect to the point of degradation?
3. What does each of the following change: prefix cache, KV 8-bit, CUDA graphs, chunk size, and max concurrent sequences?

2. Environment and methodology

The server was restarted for each test (KV cache capacity differs between initial startup and restart, refer to Section 3). Values in the tables were read from a single batch, and repetition variance was measured separately. Repeating the same test 5 times on the same server(concurrency 32, input 512, output 128) yielded throughputs of 831, 833, 831, 832, and 831 tok/s, a TTFT median of 666~670ms, a TTFT p95 of 1,606~1,608ms, and an ITL p99 of 223~224ms, remaining within 0.5%. Restarting the server 3 times also showed identical results: 66 tok/s for concurrency 1 and 831~832 tok/s for concurrency 32. Prefix cache (3 runs) and preemption/interrupts (3 runs, max ITL 227, 226, 226ms) were also identical. This means that in this environment (single GPU, no other workloads), test variance itself is small, though it may change if moved to different hardware or an environment with neighbors.

3. KV cache capacity matches the formula

The KV size per token is 2 × 층 수 × KV 헤드 수 × head_dim × 자료형 바이트. For this model with 28 layers, 8 KV heads, head_dim 128, and bf16 (2 bytes) KV, it is as follows (calculated).

2 × 28 × 8 × 128 × 2 = 114,688 바이트/토큰 (112KiB)

• The formula matches the logs. The values obtained by dividing the log memory capacity by the token count matched the formula within 0.2~0.3%. Since KV cache capacity is 「remaining memory ÷ bytes per token」, you can estimate it for any model using the numbers read from the configuration before starting the server.
• KV 8-bit packs 1.9x more tokens into the same memory. It increased from 22,048 to 41,920 tokens, and bytes per token were halved precisely.
• Capacity differed between initial startup and restarts. Despite identical options, the server initially launched in the pod had 22,048 tokens, while three servers subsequently launched in the same pod all had 26,560 tokens (~20% more). Memory used for compilation and profiling during the initial startup appears to be deducted from the KV cache allotment (estimated, not directly broken down), but when comparing capacities, you should compare bytes per tokenrather than raw memory, and record the server's startup history alongside. All tables in this post are read from the initial startup server (22,048 tokens).
• Tokens used by a single request equal the sum of input and output. With an input of 512 and output of 128, it is 640 tokens, and the 22,048-token cache accommodates about 34 requests simultaneously (calculated). The degradation point below originates from this figure.

4. Increasing concurrent requests: Throughput slows down, then the queue explodes at KV capacity limit

Input 512, output 128, --max-num-seqs 64:

• Throughput increased almost proportionally up to concurrency 8 (1 to 8, 5.8x, 73% efficiency), and slowed down thereafter(4x concurrency from 8 to 32 yielded 2.17x throughput, from 383 to 831). Overall from 1 to 32, it is 12.6x (39% efficiency). Since the benefit of amortizing weight reading costs (Parts 4 and 21) was nearly linear at 14~15x up to concurrency 16 in short contexts, the slowdown in this table is likely due to increased KV reads and input processing from accumulated 512-length inputs (estimated, not decomposed). External research also reports that attention memory reads in large batches can cause throughput to plateau well before reaching the KV capacity limit (see 「External basis」 below).
• From 32 to 64, throughput actually decreased and TTFT jumped 8.6x (from 666ms to 5.7s) (this row is from the initial startup server where KV was 22,048 tokens; repeated values measured by restarting the server are at the end of this list). If a single request uses 640 tokens, the KV cache can only hold up to about 34 requests, so sending 64 requests means nearly half of them wait in the queue until space opens up. The moment capacity (KV) hits the limit, TTFT explodes. Since throughput is already saturated, increasing concurrency further only increases latency.
• Repeating the same load across three server instances showed that these values for concurrency 64 varied by server. Throughputs were 792, 842, and 843 tok/s, with median TTFTs of 5.7, 2.1, and 2.2 seconds. Only the initially launched server was slow, and that server alone had 22,048 KV tokens (restarted servers had 26,560 tokens). Even with identical options, KV slots varied per server launch as 22,048, 22,336, 26,560, and 27,504 tokens (range across six servers with different options), and at this point where 64 concurrent requests nearly saturate the cache with 640 tokens each, a 20% increase in capacity reduced TTFT by 2.7x. This implies the above "queue explosion" is a phenomenon near this limit, and where that limit lies can vary with each server startup. Values for concurrency 8 and 32 matched within 1% across the three servers (TTFT 349, 669ms, throughput 384, 831 tok/s).
• The variation in KV slots across server startups was more pronounced in larger models. For Qwen3-4B-FP8, servers launched with identical options had KV capacities of 8,880, 13,232, and 14,448 tokens (servers with LoRa enabled had 6,320 and 10,896 tokens), and median TTFT at concurrency 32 was 7.2 seconds at 8,880 KV tokens and 3.4 seconds at 13,232 tokens. Just like the 1.7B model, the initially launched server was the smallest. Server logs' GPU KV cache size must be recorded alongside test logs to interpret values.
• Median ITL increased slowly from 15 to 25ms, while p99 jumped to about 220ms starting from concurrency 32. This happens because the step of processing new request inputs interleaves between decoding steps, causing a single token to be significantly delayed (Section 7).

Increasing the input to 2,048 tokens means a single request uses 2,176 tokens, allowing the cache to hold about 10 requests (calculated).

Multiplying concurrency by 4 from 4 to 16 increased throughput by only 37% while TTFT became 5.4x. With long inputs, the concurrency level where throughput plateaus drops significantly. This is why in operations, concurrency limits should be determined not by 「what the GPU can endure」, but by 「the number of requests KV can hold given the average input length」.

When KV is insufficient: Do they just wait, or are they evicted?

Preemption counts were read using server metrics (vllm:num_preemptions_total). During 5 repetitions of input 512 and concurrency 32 (within the ~34 request limit of KV capacity), preemption count was 0. Sending input 2,048 and concurrency 32 (64 requests) yielded 7 counts, and subsequently sending input 1,500, output 256, and concurrency 64 (128 requests) resulted in a cumulative 29 counts(22 counts under this specific load). Median TTFTs during these tests were 12.5s and 35.4s, respectively. When KV is insufficient, requests do not merely wait in the queue; running requests are evicted, relinquish their KV cache, and are processed all over again from scratch(preemption recomputes the rolled-back request, incurring an extra input processing step, per documentation). Since the region where preemption count is non-zero exhibits stagnant throughput and surging latency, it serves as a useful alerting metric.

Max concurrent sequences (--max-num-seqs) only shifts the queue location

Concurrency 64, input 512:

Lowering the upper limit to 16 decreased throughput by 23% and made TTFT 1.8x. If the upper limit is lower than the capacity (about 34), a queue forms even when capacity remains, which is a loss; even if it is higher than the capacity, there is no benefit if KV is lacking. The upper limit is a value set around the number of requests the KV capacity can hold.

5. KV 8-bit: More capacity reduces latency for long inputs

• Speeds are the same in short contexts. At concurrency 1, KV reading is small compared to weight reading, so there is no benefit from 8-bit.
• The benefit comes from capacity. Thanks to the 1.9x capacity in Section 3, queues decreased for long inputs and TTFT was significantly reduced. In other words, KV 8-bit is not about "calculating faster," but about "holding more concurrently."
• Accuracy was measured only by perplexity, and the results were counterintuitive. We sent Wikitext-2 (test split, the first 40 segments of 1,024 tokens out of about 300,000 tokens) to the server and obtained perplexity from the log probabilities of the prompt tokens (lower is better). When the same setting was measured twice, the values were identical down to the fourth decimal place (deterministic), so no separate margin of error was assigned.

• bf16 was the same regardless of the backend(17.773 to 17.780, within 0.04%). Therefore, the difference between backends is at the level of measurement error.
• fp8 KV actually lowered perplexity, and the values for the two backends differed from each other(17.42 vs. 17.03). Quantization usually adds error, so there is no reason for it to decrease. This result should not be read as "fp8 KV improves quality." The fact that results varied by 2 percentage points between backends for the same 8-bit format is a clue that numerical processing in the kernel(scale application, accumulation precision) plays a greater role, and the cause was not decomposed. Server logs warn that using fp8 without scale correction can degrade accuracy.
• To summarize, we did not obtain evidence that perplexity worsens with fp8 KV on this model and this text, but since the direction is unexplained, it cannot be used as evidence that quality is preserved either. Literature (external evidence table) reports that uncalibrated fp8 reduces inference task accuracy by up to 1-2 points depending on the model, so it should be verified with evaluation sets for one's own tasks before adoption (task accuracy was not measured).
• The initial test (bf16 4.217 vs. fp8 4.015) was invalid because the values were read from different texts, and even after correcting to the same text, the direction was the same (4.2173 vs. 4.0148), so we re-verified by fixing the backend as described above.

6. Prefix Caching: Having a common prefix makes the first token 14x faster

We attached a different 32-token tail to a 3,000-token common prefix for each request and sent them one by one in order (1 output token).

• When the cache hits, input processing for 3,000 tokens is skipped. 394ms dropped to 28ms (14x). This is the cheapest optimization for services that fix system prompts or documents at the beginning (RAG, agents).
• The cost of leaving it enabled was not visible in this test. The TTFT for inputs without prefixes was the same with enabled at 395ms and disabled at 393ms. We also measured the cost under a workload biased toward inputs with completely non-overlapping prefixes. When sending 96 requests with concurrency 32, 2,048 input tokens, and 16 output tokens 5 times, the throughput was 58 for enabled and 59 tok/s for disabled (total time 26.3–26.4s vs. 26.0–26.1s, about a 1.2% difference), and the median TTFT was 6.5–6.6s for enabled and 6.0–6.2s for disabled (about 8% longer). There were 0 hits, and the queried tokens were about 1.05 million. In other words, the cost of keeping it enabled was just over 1% throughput and single-digit percentage TTFT in input-heavy workloads (the 8% in TTFT is presumed to be an effect of requests at the front of the queue incurring more hash computations and was not decomposed).
• This is the most dangerous pitfall to watch out for when measuring. While writing this article, I measured input processing speed by repeatedly inputting the same sentence, which resulted in an amount of computation GPUs cannot possibly produce, and prefix cache hits were the cause (Episode 21). Inputs for performance tests must vary with each request. Conversely, to see cache effects, common prefixes must be intentionally created.

7. Chunked Prefill: Chunk size determines how long long inputs pause decoding

While a single decoding request (300 tokens) was running, 3 requests for 3,500-token inputs were interleaved, and the token intervals in the interleaved section were measured.

• The larger the number of input tokens processed in a single step, the longer decoding pauses accordingly. Since the input processing speed of this GPU is about 8,200 tok/s (Episode 21), the time for a single chunk is about 62ms for 512 tokens, about 250ms for 2,048 tokens, and about 430ms for 3,500 tokens (calculated). The maximum ITLs above of 69, 227, and 445ms match this calculation. The default value was verified by reading the server configuration. In this version, max_num_batched_tokens the default for is 2,048, chunked prefill is enabled (enable_chunked_prefill true), prefix caching is also enabled, and the block size is 16 tokens.
• Making chunks smaller delays input processing slightly at the cost of less interrupted decoding. The TTFT for long inputs showed a difference of about 3–5%, ranging from 481–494ms at 512 to 461–474ms at 8192. Instead, the median ITL in the interleaved section increased to 53ms at 512 (because decoding and input processing are mixed at each chunk step).
• Choose according to the use case. Small chunks are suitable for conversational and voice services where tokens must emerge seamlessly, while large chunks are suitable for batch jobs processing long documents all at once. Values between the two (1,024, 4,096) were also measured. The max ITL in the interleaved section was 129ms at 1,024 (3 iterations: 128–129) and 445ms at 4,096 (445–446). Combined with the default 2,048 (227ms), 512 (69ms), and 8,192 (445ms), the max ITL was proportional to the chunk size (1,024 tokens takes about 125ms at an input processing speed of about 8,200 tok/s). The reason 4,096 is equal to 8,192 is that the input is 3,500 tokens, making the effective chunk 3,500. The TTFT of the long input itself was nearly identical, ranging from 468–493ms at 1,024 to 461–476ms at 4,096.

8. Disabling CUDA graphs (--enforce-eager)

For a model of this size, the speed benefit of CUDA graphs was as small as about 3% (66 vs. 64 at concurrency 1), while server preparation was about 25–40 seconds faster. This is likely because, for models taking 14–15ms per token, the kernel launch interval accounts for a small proportion of the total (estimated). We noted that the benefit could be larger for smaller models, and repeating the same test 3 times with Qwen3-0.6B showed that it actually was. At concurrency 1, CUDA graph default 168 tok/s (ITL 6ms) vs. --enforce-eager 108 tok/s (ITL 9ms), graphs were 55% faster, and at concurrency 32, it was about 3.6% at 1,557 vs. 1,503 tok/s. Server preparation was 55s vs. 30s. This is likely because, with a smaller model where time per token is around 6ms, the kernel launch interval constitutes a large share of the total (estimated). Faster GPUs were not tested (not measured). In Qwen3-4B-FP8, an even larger model, the graph default at concurrency 1 was 49 tok/s vs. --enforce-eager 47 tok/s, a benefit of about 4% (ITL 19ms vs. 20ms). The benefit of graphs was large only in very small models where a single token is short, and plateaued at 3–4% for 1.7B and above.

9. Time taken to boot the server for the first time

Values read from server log timestamps.

Model download time is excluded from the above (separate). Since the time for a new pod to receive traffic in autoscaling adds image downloading and model downloading to these figures, pre-warmed replicasand model cache volumesbecome numerically necessary.

10. Takeaways for Operations

1. Concurrency limit = KV capacity ÷ tokens per request. The formula is from Section 3 and the rationale is from Section 4. Exceeding the limit does not increase throughput, only TTFT.
2. Look at TTFT p95 and ITL p99 separately. Even if the median is fine, the tail is large (ITL p99 223ms).
3. Design common prefixes intentionally. Placing system prompts and documents at the beginning and request-specific content at the end allows the prefix cache to provide a 14x benefit.
4. KV 8-bit is a capacity option, not a throughput option. It is highly effective when queues are long with long inputs. Perform quality evaluations first.
5. Chunk size is a trade-off between interruptions and input processing speed. Small chunks for conversational services, large chunks for batch jobs.
6. Performance tests should vary inputs every time and be checked against hardware limits.

11. Potential Interview Questions

• How do you estimate the maximum concurrent requests for a vLLM server? Divide the number of KV cache tokens by the average tokens per request (input + output). KV per token is 2 × 층 × KV 헤드 × head_dim × 바이트. For the 1.7B model, it is 22,048 tokens, so a 640-token request is about 34, and exceeding this caused the TTFT to jump 8x (from 666ms to 5.7s).
• Why does throughput not increase when concurrency is increased? The benefit of sharing weight reading lasts only while there is KV capacity; once capacity is full, requests wait in the queue.
• Which services benefit the most from prefix caching? Services with a long common prefix. With a 3,000-token prefix, TTFT dropped from 394ms to 28ms.
• Is the effect of KV 8-bit speed or capacity? It is capacity. Since bytes per token are halved, 1.9 times more fits into the same memory, and the TTFT for long inputs dropped from 4.0s to 0.94s. Speed for short contexts remained the same.
• Why do other users' tokens experience stuttering when a long input arrives, and how can it be reduced? The input processing phase runs in the same batch as decoding, causing a delay. Reducing the number of tokens per step via chunked prefill makes the stuttering proportional to the chunk size (up to 69ms for 512 tokens).
• How can one suspect whether performance test values are correct? Compare them against hardware limits (compute, bandwidth). If the same sentence is repeated, the prefix cache hits, resulting in impossibly high processing speeds.

External Evidence: Verified with Papers and Official Materials

All figures in this article are measured directly. We reviewed external materials reporting the same phenomenon to check the numbers and conditions, and described their relationship with our results. Since external figures differ in model, GPU, and load, we did not make direct comparisons but only judged agreement in direction and structure.

Figures from vendor and project blogs (vLLM, LMDeploy) were transcribed only when conditions were stated in the text. ACM's PagedAttention page could not be accessed, so we followed the notation on the arXiv abstract page.

Unverified Items

• Repetition variance on other hardware. Variance was within 0.5% in this environment (Section 2), but may vary in environments with different GPUs and neighbor loads. Repetitions in queues accumulating concurrency like 64 were not measured 5 times.
• Other models, other GPUs, longer contexts. Limited to a single 1.7B model, 8GB-class GPU, and inputs up to 4,096.
• --max-num-batched-tokens values between 512 and 8192 for(default 2,048 was verified).
• Output quality of KV 8-bit, and CPU overhead of prefix caching in massive requests where prefixes do not overlap.
• Whether preempted requests hit the prefix cache when resumed. We only infer that some hits may occur based on the design description that returned blocks remain in the free block queue while retaining their hashes, but this was not confirmed by official descriptions or measurements.
• The full shape of conditions under which preemption occurs. We read preemption counts via server metrics (Section 4) but did not draw curves by densely varying input/output lengths and concurrency.
• Comparison with other servers such as SGLang, multi-GPU, and quantized models (partially in Part 21).

Verified Versions and Dates

As of 2026-10-07. vLLM 0.31.0, PyTorch 2.13 (CUDA 13), Qwen3-1.7B (bf16).