Basics of LLM Inference and vLLM Internals: KV Cache, PagedAttention, and Scheduler

Status: Draft (2026-10-05), partially reflecting actual measurements. This post is written based on descriptions verified in official documentation, with direct measurements from the author's experimental environment contained in the 「Actual Measurements」 section (Section 9) and the rest based on documentation. Please read 「Verified Versions and Dates」 and 「Unverified Items」 at the end of the post first. Writing Rule: Measurement data is the baseline; unmeasured items are marked as 「Unmeasured」.

Table of Contents

• 1. Two Phases and Arithmetic Intensity
• 2. KV Cache and Sizing Formulas
• 3. From Static Batching to Continuous Batching
• 4. PagedAttention and Block Management
• 5. Prefix Caching
• 6. Chunked Prefill, Scheduler, and Preemption
• 7. Six Operational Parameters
• 8. Metrics and /metrics
• 9. Actual Measurements: LRU Collapsed under Cyclic Access
• 10. Try It Yourself
• 11. Likely Interview Questions
• 12. Common Misconceptions
• References
• Verified Versions and Dates, and Unverified Items

The throughput of LLM serving is determined by two things rather than raw compute speed: how much KV cache can fit into the GPU memory, and how many requests share the weights read when generating a single token. This post explains the reasons using arithmetic intensity, and summarizes PagedAttention's block management, prefix caching, chunked prefill, preemption, operational parameters, and metrics as verified in the source code and official documentation of vLLM v0.30.0. The author has no experience operating large-scale production serving and only has experience measuring on consumer GPUs.

What You Will Be Able to Explain in an Interview After Reading This Post

1. You can state that prefill is compute-bound and decode is memory-bandwidth bound using arithmetic intensity, and thus explain why batching and KV capacity determine throughput.
2. You can calculate KV cache size using formulas and explain how GQA, blocks (PagedAttention), prefix caching, and preemption utilize that capacity.
3. You can link six operational parameters with /metrics to explain the sequence for diagnosing bottlenecks.

1. Two Phases and Arithmetic Intensity

Inference is divided into prefill, which processes the entire prompt at once to populate the KV cache, and decode, which generates tokens one by one. Arithmetic intensity is the number of operations performed per byte read from memory (FLOP/byte). When the number of parameters is P and weights are 2 bytes, the computation volume of a linear layer processing n tokens at once is approximately 2·P·n FLOPs, and since weights are read only once, the amount read is approximately 2·P bytes. Therefore, the intensity is approximately n FLOP/byte (an approximation excluding attention and KV reads).

Prefill is compute-bound because n is the prompt length. For decode, since a single request has n=1, the number of concurrently processed requests B acts as n. If B is smaller than the value obtained by dividing the GPU's compute performance by its memory bandwidth (the ridge point), it is memory-bandwidth bound. Using illustrative examples of 3.35 TB/s and 1,979 TFLOPS for FP16 Tensor Cores (noted with sparsity) as specified by NVIDIA for the H100 SXM, and assuming dense performance is half of the specification, the ridge point is approximately 295 FLOP/byte. Unless concurrent requests reach the hundreds, decode time is dominated by the time spent reading weights.

According to the author's experimental environment logs, on a laptop GPU with a memory bandwidth of 896 GB/s (NVIDIA specification), Qwen3.8-27B NVFP4 showed 39 tok/s for a single request with a load size of 16.2GB, and a combined 138 tok/s for 4 concurrent requests. 39 × 16.2GB is approximately 632 GB/s, which is about 70% of the rated bandwidth, and the fact that the combined rate is about 3.5x is naturally viewed as the result of four requests sharing the weights read per step. This is an estimation treating the load size as the amount read per step, and the prompt and output lengths at the time of measurement were not recorded, so they could not be verified.

2. KV Cache and Sizing Formulas

Attention revisits the keys and values of all preceding tokens, so they are stored per layer. The size per token is as follows:

토큰당 바이트 = 2(K와 V) × 층 수 L × KV 헤드 수 H_kv × head 차원 d × 원소 바이트 b

The values were verified from Hugging Face's config.json (num_hidden_layers, num_key_value_heads). Since they are all BF16, b=2, and head dimension 128 is calculated as head_dim because there is no hidden_size ÷ num_attention_heads in the configuration.

MQA (Multi-Query Attention, Shazeer 2019) has all query heads share a single K and V (H_kv=1), and GQA (Grouped-Query Attention, Ainslie et al. 2023) is an intermediate approach that groups query heads to have one per group. The GQA paper claims that additionally training MHA checkpoints with 5% of the original pretraining computation yields speeds close to MQA and quality close to MHA. A smaller H_kv increases the number of requests that can be accommodated and reduces the bytes read during decode, and storing KV in FP8 (--kv-cache-dtype fp8) makes b=1, halving it again.

The formula was aligned with the author's experimental logs. Qwen2.5-1.5B is 28,672 bytes per token, and the LMCache log "4,864 tokens, size 0.1299 gb" matches 4,864 × 28,672 ÷ 2^30 = 0.12988 (when reading gb as GiB). By the same token, 21,520 tokens on the GPU is about 0.575 GiB, but the available KV memory log line from the server was not saved, so it could not be directly compared.

3. From Static Batching to Continuous Batching

Static batching starts and ends a batch of requests together, leaving the space freed by a shorter request vacant until the longest request finishes. Orca (OSDI 2022) changed the scheduling unit from a request to an iteration (generating a single token), allowing finished requests to be removed and new requests to be added at every iteration. This is continuous batching, and the paper claims a 39.9x throughput increase on GPT-3 175B compared to FasterTransformer at the same latency (figures under paper conditions). The comments in the vLLM V1 scheduler source generalize this further: there are neither decoding phases nor prefilling phases, and tokens are simply allocated for each request so that num_computed_tokens catches up with num_tokens_with_spec , and notes that chunked prefill, prefix caching, and speculative decoding are all handled through this expression.

4. PagedAttention and Block Management

Traditional servers pre-allocated contiguous memory up to the maximum length per request, and in Figure 2 of the paper (average memory waste ratio from experiments in Section 6.2), the proportion of KV memory actually used for token states in the three Orca variants implemented directly by the authors (Max, Pow2, Oracle) was 20.4–38.2%, while vLLM reads at around 96%. Since Orca's public implementation does not exist, the variants are the authors' implementations, and this ratio cannot be generalized to the values of all traditional servers. The throughput improvement in the same paper (1.7–2.7x compared to Orca Oracle, 2.7–8x compared to Orca Max) is also based on these comparison targets. PagedAttention (SOSP 2023) divides KV into fixed-size blocks (default 16 tokens) like virtual memory in an operating system, maintains a block table mapping logical blocks to physical blocks per request, and allocates physical blocks one by one as needed. Waste is reduced to the empty space in the last block per request, and identical prefixes can share a single physical block (reference counting and copy-on-write).

요청 A 논리 블록        block table(A)   물리 블록 풀(GPU)
B0 [토큰  0-15]    ->   물리 7     ---+
B1 [토큰 16-31]    ->   물리 1        |  물리 7 : ref_cnt=2 (A, B 가 공유)
B2 [토큰 32-37]    ->   물리 3        |  물리 3 : 6/16 만 참  <- 낭비는 여기뿐
요청 B: B0 -> 물리 7 -----------------+  (같은 접두사)

The performance in the paper is distinguished as claims. The paper notes that when measured on an A100 with OPT 13B/66B/175B and LLaMA 13B using ShareGPT and Alpaca traces, throughput was 2 to 4 times higher than FasterTransformer and Orca at the same latency, and the attention kernel was 20–26% slower than FasterTransformer due to traversing the block table. The paged_attention section of the vLLM design document states that it is a historical document based on the original paper and does not describe today's code, so while the concept remains valid, the kernel implementation has changed. The block pool (source v0.30.0) manages free blocks as a doubly linked list queue and reserves block 0 as a null block. The 21,520 tokens in the experimental log correspond to 1,345 blocks assuming the default block size of 16, of which 1,344 blocks are usable.

5. Prefix Caching

Block hashes form a chain created from the parent block's hash, this block's tokens, and additional keys (LoRA ID, multimodal input hash, cache_salt). Since even a single different token in the beginning changes all subsequent blocks, only prefixes hit. Only full blocks are cached, so hits are multiples of 16 tokens, and even with a full hit, the last token is recomputed to obtain logits (source comments). When a request finishes, blocks are returned to the free queue while retaining their hashes, and are reused via LRU from the front of the queue, with blocks returned from the back so that tail blocks are evicted first (source comments). The reasoning that earlier blocks are more likely to be reused is my interpretation, not explicitly stated by the source.

In v0.30.0, enable_prefix_caching is enabled by default if supported by the model. According to the V1 blog, V0 had high CPU overhead when hit rates were low, so it was disabled by default, whereas V1 claims almost no degradation even at 0% hits. Prefix caching only reduces prefill time and cannot reduce decode time (official documentation). In multi-tenancy, trust groups are separated using cache_salt .

6. Chunked Prefill, Scheduler, and Preemption

The scheduler sets a token budget per step (max_num_batched_tokens). The request order is as follows: tokens are first assigned to running requests (decode and ongoing prefill chunks), new requests are accepted from the front of the queue using the remaining budget, and prompts longer than the budget are truncated to fit only partially (chunked prefill, enabled by default in the official documentation for all applicable cases). The goal is to stabilize the delay between tokens by preventing a single long prefill from blocking the decode of other requests for multiple steps. The Sarathi-Serve paper (2024) uses this idea to claim a serving capacity 2.6 times that of vLLM at the time under a latency SLO on a single Mistral-7B/A100.

flowchart TD
  A[스텝 시작: 토큰 예산 = max_num_batched_tokens] --> B[실행 중 요청에 토큰 배정]
  B --> C{새 블록을 받을 수 있는가}
  C -- 예 --> D[남은 예산으로 대기열 앞에서 새 요청 받기]
  C -- 아니오 --> E[가장 늦게 들어온 실행 요청 선점: 블록 반납, 대기열 맨 앞으로]
  E --> C
  E -.-> F[선점이 있던 스텝은 새 요청을 받지 않음]
  D --> G[긴 프롬프트는 예산만큼 잘라 넣고 GPU 한 스텝 실행]

If KV blocks are insufficient, preemption occurs. The source code (FCFS policy) selects the request at the very back of the execution list—that is, the one that arrived most recently—returns all of its blocks, and resets num_computed_tokens back to 0, placing it at the front of the queue. This is recomputation. The official documentation also explains that in V1, recompute is the default rather than swap, and it incurs lower overhead. Because preemption repeats work twice, it harms both TTFT and throughput. The mitigations recommended by the documentation are to raise gpu_memory_utilization , reduce max_num_seqs or max_num_batched_tokens , and increase tensor_parallel_size .

7. Six Operating Parameters

Default values are based on the v0.30.0 source code and may vary by version. Providing --performance-mode throughput doubles two default values that were not explicitly specified by the user (source).

The author's experimental deployment configuration is --gpu-memory-utilization=0.93 --max-model-len=16384 --max-num-seqs=4 --max-num-batched-tokens=2048 --kv-cache-dtype=fp8 , and based on actual measurements noted in the configuration, the KV cache is 2.9GB (approx. 57,000 tokens). Since this size accommodates about 3.5 requests of 16,384 tokens, the possibility that preemption could occur if all 4 concurrent requests become long is an arithmetic estimation, not an observed fact.

8. Metrics and /metrics

TTFT is the time from sending a request until the first output is received, TPOT is (total latency - TTFT) ÷ (number of output tokens - 1), and ITL is the interval between streaming outputs. In speculative decoding, a single output contains multiple tokens, which makes ITL and TPOT diverge (vLLM benchmark documentation). Goodput is the throughput counting only requests that satisfy SLOs such as TTFT and TPOT (DistServe paper, OSDI 2024). We look at p50, p95, and p99 instead of the average.

These are metrics verified in the v0.30.0 source code (Prometheus counters are appended with _total ).

• vllm:num_requests_running, vllm:num_requests_waiting: An increase in waiting is a saturation signal.
• vllm:kv_cache_usage_perc: Values close to 1 indicate KV saturation. vllm:num_preemptions_total increases when recomputation is underway.
• vllm:prefix_cache_queries_total, vllm:prefix_cache_hits_total: Hit rate is hits ÷ queries.
• vllm:time_to_first_token_seconds, vllm:inter_token_latency_seconds, vllm:request_queue_time_seconds, vllm:request_prefill_time_seconds, vllm:request_decode_time_seconds: Breaks down latency into waiting, prefill, and decode.

If waiting increases, kv_cache_usage_perc is close to 1, and preemption increases, it indicates a KV shortage. If waiting increases but utilization is low, max_num_seqs or the token budget is the bottleneck. Because histograms are divided by bucket boundaries (TTFT uses 0.5, 0.75, 1.0 seconds, etc.), the p99 of histogram_quantile is as coarse as the bucket width.

9. Actual Measurement: LRU Broke Down in Cyclic Access

These are actual measurements taken by the author. A single 8GB-class laptop GPU, image lmcache/vllm-openai:latest(vLLM 0.29.0, LMCache 0.5.5), model Qwen2.5-1.5B-Instruct, --gpu-memory-utilization 0.60 --max-model-len 8192 --max-num-seqs 4 --enable-prefix-caching . The server log showed the GPU KV cache was 21,520 tokens. Ten documents composed of random words without overlapping prefixes (each approx. 5,011 tokens, totaling approx. 50,000 tokens) were read sequentially, read again in the same order, and then read in reverse order, with requests sent one by one.

Because the total sum of the documents is 2.3 times the GPU capacity, reading them again in the same order under LRU results in each document being evicted right before the moment it is needed, yielding 0 hits. The 21,488 token hits in the third pass (reverse reading) correspond to 1,343 blocks (×16) out of 1,344 available blocks. This explanation was verified using the block LRU model in Section 10, which also produced 0 for pass 1, 0 for pass 2, and 21,488 for pass 3. The fact that the model matched the actual measurements is evidence that the explanation is consistent, not a strict proof.

When LMCache offloaded the KV to CPU memory, the second pass TTFT was 48.6ms. Experimental logs recorded that loading 4,864 tokens per document from the CPU took about 8 to 9ms based on log timestamps, while recomputing the same tokens from scratch took about 560ms. During the initial read, storage overhead increased TTFT p50 by about 4%, from 575.5 to 597.8ms.

Limitations are disclosed as-is. Since concurrency was 1, situations where requests overlap were not measured. Because the model is 1.5B, the GPU is a single unit (8GB), and the documents are synthetic, the expectation in the experimental notes that the benefits would be greater with a larger model is not demonstrated by this measurement. These are single-measured values with no standard deviation, and with 10 requests per pass, p95 is practically the maximum value. The gains came from CPU offloading rather than PagedAttention, representing an optimal scenario where prefixes are heavily repeated. After the experiment, we checked the vLLM documentation's own OffloadingConnector , but did not re-measure under the same conditions.

10. Try It Yourself

Steps 1 and 2 involve local computation or read-only operations.

# 1) 클러스터 없이: KV 크기 계산
curl -sL https://huggingface.co/Qwen/Qwen2.5-7B-Instruct/resolve/main/config.json | python3 -c '
import json,sys; c=json.load(sys.stdin)
d=c["hidden_size"]//c["num_attention_heads"]
b=2*c["num_hidden_layers"]*c["num_key_value_heads"]*d*2
print(b, "바이트/토큰", b*32768/2**30, "GiB/32K 토큰")'

# 2) 읽기 전용 확인(vLLM 서버가 떠 있는 클러스터에서. 서버를 새로 켜는 일은 변경이라 하지 않는다)
kubectl logs <vLLM 파드 이름> | grep -E "GPU KV cache size|Maximum concurrency"
kubectl exec <vLLM 파드 이름> -- python3 -c "import urllib.request as u;print(u.urlopen('http://127.0.0.1:8000/metrics').read().decode())" | grep -E "^vllm:(num_requests|kv_cache_usage|num_preemptions|prefix_cache)"

Step 3 is a block LRU model. This is a simplified model of my own creation, not actual measurements (16 tokens per block, tail-first eviction, no hashing for partial blocks).

from collections import OrderedDict
BS, TOTAL, DOCS, N = 16, 1344, 10, 5033   # 쓸 수 있는 블록 = 전체 1345 - null block
free, fresh = OrderedDict(), TOTAL        # free: 해시가 붙은 빈 블록, 앞쪽이 먼저 방출됨

def serve(d):
    global fresh
    hs = [(d, i) for i in range(N // BS)]            # 가득 찬 블록의 해시
    hit = 0
    while hit < len(hs) and hs[hit] in free and (hit + 1) * BS <= N - 1:
        hit += 1
    for h in hs[:hit]:
        del free[h]                                  # 적중한 블록은 사용 중
    for _ in range(-(-N // BS) - hit):               # 모자란 블록을 새로 받는다
        if fresh: fresh -= 1
        else: free.popitem(last=False)               # 가장 오래된 블록을 내보냄
    for h in reversed(hs):
        free[h] = 1                                  # 끝나면 꼬리부터 큐에 넣는다
    fresh += N % BS > 0                              # 부분 블록은 바로 재사용
    return hit * BS

for name, order in (("1차", range(DOCS)), ("2차", range(DOCS)), ("3차 거꾸로", reversed(range(DOCS)))):
    print(name, sum(serve(d) for d in order))

Step 4 is a temporary pod reproduction. The original experiment ran using a temporary pod with 1 GPU and 6Gi of memory. The risks are memory contention with other workloads on the same node and GPU occupation. Pods are configured to self-terminate after 1 hour even if left running, and object deletion is performed manually. This article did not re-execute this reproduction.

11. Potential Interview Questions

1. Why is decode memory-bandwidth-bound? When a single request generates a single token, it reads all weights, but there is only about one multiplication per weight, making the arithmetic intensity comparable to the concurrent request count B. This is smaller than the ridge point (several hundreds on H100), leaving compute units idle, and batching—which groups requests to share weight reads—increases throughput.
2. Calculate the KV cache size and explain the effect of GQA. 2 × number of layers × number of KV heads × head dimension × bytes is the size per token. For Qwen2.5-7B, it is 28 × 4 × 128, which is 56KiB. If it were MHA where all 28 query heads possessed KV, it would be 392KiB (7 times larger), so GQA correspondingly increases the number of requests that can be accommodated simultaneously.
3. What problem does PagedAttention solve, and how is the block size determined? It eliminated the fragmentation and waste of the previous approach of pre-allocating contiguous memory for the maximum length by using dynamic block-level allocation. Larger blocks improve kernel efficiency but increase waste in the final block, while smaller blocks do the reverse; the paper chose 16 as the default to balance this trade-off.
4. What does vLLM do when the KV cache becomes full, and how can you tell? It preempts the running request that arrived most recently, returns its blocks, and later recomputes from scratch. kv_cache_usage_perc approaches 1, num_preemptions_total increases, and the queue grows longer. Countermeasures include raising utilization, shrinking max_num_seqs , using KV FP8, and adding tensor parallelism.
5. Prefix caching is enabled, but hits are not occurring. What should I check? prefix_cache_hits_total ÷ queries_total Check the actual hit rate with

12. Common Misconceptions

• "PagedAttention is an algorithm that speeds up attention computation." It is a memory management technique. The paper also states that kernels became 20–26% slower, and the gains come from running larger batches with waste-free memory.
• "--gpu-memory-utilization "Raising makes things faster."
• "Prefix caching speeds up the entire response." It only reduces prefill, so the benefit is small for workloads with long outputs.
• "The 12x figure from the experiment represents vLLM's performance." It is the gain achieved by LMCache's CPU offloading in an optimal scenario, measured once with concurrency 1 on a 1.5B model.
• "A low average TTFT is sufficient." Service quality is determined by p99, and averages hide the tail stretched out by preemption.

References

Only items that were actually opened and examined are listed.

• Papers: PagedAttention https://arxiv.org/abs/2309.06180 (body verified via ar5iv), Orca https://www.usenix.org/conference/osdi22/presentation/yu, Sarathi-Serve https://arxiv.org/abs/2403.02310, DistServe https://arxiv.org/abs/2401.09670, GQA https://arxiv.org/abs/2305.13245, MQA https://arxiv.org/abs/1911.02150
• vLLM V1 Blog: https://vllm.ai/blog/2025-01-27-v1-alpha-release
• vLLM Documentation: https://docs.vllm.ai/en/latest/ under configuration/optimization, design/prefix_caching, design/paged_attention, design/metrics, benchmarking/cli, features/kv_offloading_usage, serving/parallelism_scaling
• vLLM v0.30.0 Source (https://github.com/vllm-project/vllm/tree/v0.30.0): vllm/config/cache.py, config/scheduler.py, engine/arg_utils.py, v1/core/sched/scheduler.py, v1/core/block_pool.py, v1/core/kv_cache_manager.py, v1/metrics/loggers.py
• Model configurations: https://huggingface.co/Qwen/Qwen2.5-1.5B-Instruct plus 7B and 72B. NVIDIA: https://www.nvidia.com/en-us/data-center/h100/ , https://www.nvidia.com/en-us/geforce/laptops/50-series/

Verified versions and dates, and unverified items

• The verification date is 2026-10-05. The latest stable release of vLLM is v0.30.0 (2026-09-22), and the source and defaults are based on this tag. The author's experiments were run on v0.29.0. In v0.29.0, Model Runner V2 became the default for all models, and the V1 runner is deprecated with plans for removal in v0.32 (release notes).
• Unverified items: H100 dense FP16 figures (NVIDIA's page only includes sparsity-inclusive values, so assumed to be halved), measurement conditions for 27B actual measurements, the fact that the log gb is in GiB (estimated by numerical match), the block size of the experimental server (assumed to be 16 by default), log verification for why the third hit is 1,343 blocks (explained by model only), and the exact format of the option to disable prefix caching.