KV Offloading and Disaggregated Serving: Reloading Evicted Prefixes from the CPU

Status: Draft (2026-10-27), offloading measured, disaggregated serving unmeasured. Measured on 2026-10-27 using a single 8GB laptop GPU (Ada generation), vLLM 0.31.0, and Qwen3-Reranker-0.6B. The principles and measurements of offloading and disaggregated serving using LMCache and NVIDIA Dynamo are covered in Parts 3 and 4 of "Anatomy of LLM KV Cache", so this post focuses on vLLM's built-in CPU offloading. Writing rules: Measured data takes priority; unmeasured items are marked as "Unmeasured", and unverified items as "Unverified".

Table of Contents

• 1. Prefix Caches Inside the GPU Are Quickly Evicted
• 2. vLLM's Default Offloading
• 3. Measuring It Yourself
  • Method
  • Results
• 4. When Is There No Benefit?
• 5. Disaggregated Serving: Sending KV to Another Engine via the Same Connector
• 6. Takeaways for Production
• 7. Unverified Items
• References

1. Prefix Caches Inside the GPU Are Quickly Evicted

Part 2 showed that reusing a 3,200-token prefix reduces the time to the first token from 174.7ms to 18.9ms. However, that block can only be reused if it remains in the "free space" of the GPU KV cache. When a new request demands space, vLLM evicts the oldest unused blocks first. In a service querying multiple long documents alternately, a few gigabytes of GPU memory will fill up quickly.

KV offloading is a feature that copies blocks to a larger, slower medium—specifically CPU memory—before eviction (strictly speaking, when a block becomes full). When the same prefix appears again, any blocks missing from the GPU are retrieved from the CPU and restored. The break-even point—where transferring is faster than recomputing—was covered in the LMCache post.

2. vLLM's Default Offloading

vLLM bundles the exchange of KV data outside the engine into a single interface called the KV Connector. Multiple implementations attach to this same interface.

Default offloading is enabled with two lines. --kv-offloading-size specifies the CPU buffer size in GiB, and providing this value activates the native backend (specifically OffloadingConnector).

vllm serve <모델> --kv-offloading-size 0.5 --kv-offloading-backend native

Detailed configurations are passed via --kv-transfer-config 's kv_connector_extra_config . Based on the 0.32.0 documentation, options include CPUOffloadingSpec(default), which uses a single CPU, and TieringOffloadingSpec , which places a secondary tier like a file system beneath the CPU. Eviction order selects either lru or arc . By default, only prompt (prefill) blocks are offloaded, while decode blocks are not (offload_prompt_only). The documentation notes that copying between GPU and CPU overlaps with computation via cudaMemcpyAsync .

3. Measuring It Yourself

Method

An eviction scenario was intentionally simulated.

• The GPU KV cache was restricted using --kv-cache-memory-bytes 268435456(256MiB) so that only 2,336 tokenscould fit. --max-model-len 2048.
• Each batch consisted of five requests. Prefix A (approx. 1,600 tokens) was sent first (cold), immediately sent a second time (GPU hit), followed by other prefixes B and C to evict A, and finally prefix A was sent again.
• Five batches were run on both a server without offloading and a server with --kv-offloading-size 0.5(approx. 4,600 token capacity). The first request of the first batch is viewed separately due to server warmup effects.
• The rest of the environment matches Part 2 (8GB laptop GPU, vLLM 0.31.0, Qwen3-Reranker-0.6B, container memory limit 4GiB).

Results

Looking at all five batches, "A Resent" on the offloading server clocked in at 35.3, 35.4, 34.8, 33.3, and 35.6ms, consistently hovering around 34ms, whereas the server without offloading matched cold requests at 76.3–78.0ms. The very first request on both servers reached 104ms, absorbing nearly 30ms of warmup overhead.

Server metrics were also checked to verify what was transferred.

Here is how to read the data:

1. The time to the first token for the evicted prefix became 2.2x faster (77.3ms → 35.1ms). However, it is still 21ms slower than when it remains in the GPU (14.0ms). Most of that difference comes from the 13.5ms required to transfer 170MB.
2. The KV transfer speed on this GPU was approximately 12.6GB per second. Recomputing the same 1,480 tokens takes about 63ms (calculated by subtracting the GPU hit time of 14.0ms from 77.3ms). With a computation speed of roughly 23,000 tokens per second and a transfer speed of roughly 110,000 token equivalents per second, transferring is nearly five times faster in this combination.
3. Offloading is not free. A total of 2.82GB was offloaded, including blocks that were never reused. Because copying is asynchronous, the difference was barely noticeable in the cold request TTFT of this measurement (76.8 vs 77.7ms), but it continuously consumes PCIe bandwidth and CPU memory.
4. The CPU buffer uses pinned memory. Allocating 0.5GiB locks those pages, counting directly against the container memory limit. This test used a small buffer to stay within the 4GiB limit.

4. When Is There No Benefit?

• The smaller the model and the faster the GPU computation, the lower the benefit. This is because recomputation time approaches transfer time. Conversely, larger models increase computation per token while KV bytes per token grow more slowly thanks to GQA and MLA. In Part 3 of "Anatomy of LLM KV Cache", LMCache measurements on the same GPU (576ms → 48.6ms) showed a greater benefit than this test because the inputs were longer.
• Without reuse, offloading only incurs costs. Since all prompt blocks are offloaded, workloads where prefixes do not reappear consume PCIe bandwidth and pinned memory for nothing. This can be reduced using store_threshold(offloading only blocks accessed a certain number of times) (unmeasured).
• There is no use for offloading when GPU space is plentiful. The default server in Part 2 held 40,208 KV tokens, meaning prefix A was never evicted in the same test.

5. Disaggregated Serving: Sending KV to Another Engine via the Same Connector

Disaggregated serving, which places prefill and decode on separate GPUs, operates on top of the connector architecture described above. The prefill engine generates KV, sends it to the decode engine via NixlConnector , and the decode engine continues generation using the received KV. The vLLM documentation cites two main reasons for this:

• To tune TTFT and ITL independently (allowing different parallelism for prefill and decode)
• To control tail ITL. While the "long input pausing decode" issue observed in Part 2 can be mitigated with chunk sizes, the documentation notes that selecting the right chunk size is difficult.

The same documentation explicitly states that "disaggregated serving does not increase throughput" and labels the entire feature as experimental. The original paper, DistServe (OSDI 2024), reported 7.4x to 12.6x stricter request goals compared to traditional systems when evaluated by maximum request rate per GPU while satisfying both TTFT and TPOT targets (based on the abstract). Version 0.32.0 also includes a feature where the decode side reuses token IDs generated by the prefill side (kv_transfer_params 's prompt_token_ids).

Because I only have a single GPU, standalone vLLM disaggregated serving remains unmeasured here. Results from running disaggregated serving with NVIDIA Dynamo across two machines connected via 1GbE and a single GPU family (5x slower on a single card, over 20x slower between two machines) can be found in the Dynamo post and Part 24 of "Toss ML/AI Infra Preparation". The conclusion that KV transfer must be as fast as NVLink or RDMA to be meaningful aligns with this offloading measurement (12.6GB/s over PCIe).

6. Takeaways for Production

• For services querying multiple long documents alternately (RAG, code repository querying, long conversations), CPU offloading cuts the TTFT of evicted prefixes by more than half with a single configuration option. It is worth turning on first when GPU KV utilization (vllm:kv_cache_usage_perc) is consistently high and prefix hit rates are lower than expected.
• Once enabled, use vllm:prompt_tokens_by_source_total{source="external_kv_transfer"} and vllm:kv_offload_load_time_total to verify whether data is actually being loaded and how long it takes.
• The CPU buffer size adds directly to the container memory limit. Pinned memory cannot be swapped, and exceeding the limit will cause the engine to crash with an OOM.

7. Unverified Items

• TieringOffloadingSpec The file system tier of arc policy, and store_threshold are unmeasured.
• The impact of offloading copy operations on decode throughput under high concurrency is unmeasured. This test sent requests sequentially.
• Standalone vLLM disaggregated serving (NixlConnector) is unmeasured due to having only a single GPU.

References

• vLLM Documentation, KV Offloading Usage Guide, Disaggregated Prefilling (0.32.0): https://docs.vllm.ai/en/latest/features/kv_offloading_usage.html, https://docs.vllm.ai/en/latest/features/disagg_prefill.html
• vLLM 0.11.0, 0.19.0 Release Notes (CPU Offloading): https://github.com/vllm-project/vllm/releases
• Zhong et al., "DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving", OSDI 2024. https://arxiv.org/abs/2401.09670
• Previous post: LMCache: Moving KV Cache Out of the GPU with NVIDIA Dynamo