Status: Draft (2026-10-13), partial measurements. Measurements re-cite values measured with an 8GB-class GPU, Qwen2.5-1.5B-Instruct, vLLM 0.29.0, and LMCache 0.5.5 (in-process mode) from Part 8 of "Toss ML/AI Infra Preparation" on the same site. The paper and official documentation were opened and verified directly on 2026-10-13. LMCache's currently recommended approach (MP mode) and the values for larger models are unmeasured (awaiting approval because new images and models need to be downloaded). Added on 2026-10-27. Upon approval, we measured MP mode using the official vLLM 0.32.0 image (which already includes LMCache 0.5.6) and Qwen3-0.6B. The results are in Section 9, while the 2026-10-13 figures from the preceding sections have been left as-is. Large models and multi-server setups remain unmeasured. Writing rules: Measured data is the baseline; unmeasured items are marked as "Unmeasured", and unverified items are marked as "Unverified".

LMCache: Moving KV Cache Out of the GPU

Table of Contents

• 1. Two Limitations of Cache Inside the GPU
• 2. Architecture of LMCache
• 10. Why TTFT and Throughput Improve
• 4. Directly Measured Values: From 576ms to 48.6ms
• 5. Break-Even Point: Only When Retrieval is Cheaper than Computation
• 6. Reusing Non-Prefix Chunks Too: CacheBlend, CacheGen
• 7. KV Transfer in Prefill-Decode Disaggregation
• 8. When There is No Benefit
• 9. Re-measurement of MP Mode on 2026-10-27
  • 9-1. Environment and Methodology
  • 9-2. Basic Effect: Re-reading Document Chunks That Do Not Fit in the GPU
  • 9-3. By Length: Recomputation vs. Retrieval
  • 9-4. Capacity: Partial Hits Instead of a Cliff, and the Reverse Effect of Zipfian Distribution
  • 9-5. Advantages Unique to MP Mode
  • 9-6. L2 Local Disk Tier and Break-Even Point
  • 9-7. Metrics
  • 9-8. Summary
• References

1. Two Limitations of Cache Inside the GPU

The prefix cache from Part 2 is powerful, but it only works inside GPU memory. Because of this, it has two regrettable limitations.

1. It gets evicted when space is tight. When an evicted prefix reappears, it is computed from scratch.
2. It is isolated per instance. When the same model is deployed across multiple GPUs, neighboring instances cannot use the KV just computed by another instance.

We once measured how painful the first limitation is. A document bundle (10 documents, about 5,000 tokens each, totaling about 50,000 tokens) larger than the GPU's KV capacity (21,520 tokens) was fed sequentially and then read again in the same order . When using vLLM alone, the first token latency of the second read was 576ms, identical to the first (575ms), and vLLM's strong prefix cache hit rate was 0. This is because under a least-recently-used (LRU) eviction policy with cyclic access, even a slight shortage in space causes the document needed in the next turn to always have been just evicted.

2. Architecture of LMCache

LMCache is a KV cache tierattached alongside the serving engine. The architecture verified in the technical report (arXiv 2510.09665, Dec 2025 revision) and official documentation (viewed 2026-10-13) is as follows.

• The touchpoint with the engine is the KV connector. vLLM provides a standard interface (KV connector) to export and import KV outside the engine, and LMCache is one of its implementations. The scheduler side asks, "How many tokens of this request's prefix were found externally?", and the model execution side initiates and awaits loading and saving per layer.
• The unit is the chunk. By default, 256 tokens are handled as a single block. The reason for grouping them larger than vLLM's block (16 tokens) is that storage and networks are slow when transferring many small fragments. The report cites other measurements as a basis that the transfer unit must be large to fully utilize bandwidth.
• The storage hierarchy is GPU → CPU memory → local disk → remote storage. Redis/Valkey, S3, Mooncake, NIXL, InfiniStore, shared file systems, and GPUDirect Storage are listed as backends in the report and documentation.
• Mechanisms to prevent waiting. Saving is done asynchronously in the background. Loading overlaps with computation layer by layer (fetching the next layer's KV while computing the current layer). While a request is in the queue, KV from slower tiers may also be pre-fetched into faster tiers.
• **The currently recommended deployment is a separate server running once per node (MP mode)**. Multiple vLLMs share one lmcache server . The initial "method running inside the engine process" is marked as legacy and deprecated in the documentation. The latest stable version is v0.5.5 (2025-09-12), and a v0.5.6 pre-release is available.

Because the same prefix is used as a key, KV saved by instance 1 can be found and used by instance 2 . If they are on the same node, it uses the CPU memory of the MP server; if on different nodes, it uses a P2P approach reading neighbor memory via shared storage or RDMA.

3. Why TTFT and Throughput Improve

The core can be summarized in one line: Replacing recomputed prefills with storage retrieval.

• TTFT. Time to first token is predominantly prefill (Part 1). If you load the first 4,864 tokens, you only need to compute the remaining ~150 tokens.
• Throughput. Prefill is the phase that consumes the most GPU compute. Skipping it allows the same GPU to accept more requests in the same amount of time. Consequently, the "request rate handled while maintaining the same TTFT" increases.
• Inter-token latency (ITL). As seen in Part 2, long prefills stall decoding on the same GPU. Reducing prefills also reduces interruptions, making token intervals more uniform.

Figures from the report (8×H100 server, six models including Llama-3.1-8B/70B and Qwen2.5-72B, CPU offload limit 500GB):

The abstract mentions throughput "up to 15x", but the maximum confirmed in the results section was 14x from Section 8.2. The remote storage benefit being 1.3–3x, smaller than CPU memory, is because the link is slower, which is the same story as the break-even point in the next section.

4. Directly Measured Values: From 576ms to 48.6ms

We turned on LMCache for the experiment in Section 1 (CPU memory limit 1.5GB, chunk 256). The environment was an 8GB laptop GPU, Qwen2.5-1.5B-Instruct (fp16), vLLM 0.29.0, LMCache 0.5.5, with requests sent one by one.

"Again" refers to the second pass read again in the same order.

• The first token latency of the re-read became about 12x faster. The LMCache logs showed that out of roughly 5,000 tokens per request, 4,864 tokens (256 × 19) were loaded from the CPU, and the number of tokens newly computed by the engine was 0. Due to chunk-based granularity, the final deficient fragment could not be loaded and was recomputed.
• The loaded KV was 0.13GB per document. Using the formula from Part 1, Qwen2.5-1.5B (28 layers, 2 KV heads, head dimension 128, 2 bytes) has 28,672 bytes per token, so for 4,864 tokens it is 0.1299 GiB, which matches the logs.
• The initial read was about 4% slower(575.5 → 597.8ms). This is the cost of offloading KV to the CPU while reading.
• GPU memory usage was identical between the two settings. LMCache does not consume additional GPU space.

Varying the length shows that the benefit increases with context length (CPU 2GB, re-requesting documents evicted from the GPU).

While recomputation time is almost proportional to length, loading time grows gently, taking only a few ms longer than a direct GPU hit (20–32ms).

Capacity is a cliff. We measured the same re-read while varying only the CPU limit (the working set's KV was about 1.49GB by computation).

Even if it was slightly smaller than the working set, the benefit was 0, leaving only the storage cost. This is the trap of LRU cyclic access, same as in Section 1. Under a realistic distribution where some documents are read frequently (Zipfian distribution, 60 documents), even a small cache helped, reducing average first token latency from 278ms with vLLM alone to 174ms for CPU 1GB and 63ms for CPU 2GB.

Limitations of this experiment: 1 concurrency, a single 1.5B model, synthetic documents made of random words, single-measurement values, and using the in-process mode rather than the MP mode recommended by the current documentation.

5. Break-even: When loading is cheaper than recomputing

For loading to be beneficial, the time to transfer the KV cache must be shorter than the time to recompute that KV cache. For n tokens, it can be written as follows:

• Recomputation: n × (prefill time per token)
• Loading: n × (KV bytes per token) ÷ bandwidth + fixed latency

Since n is on both sides, subtracting the fixed latency gives: Break-even bandwidth = KV bytes per token ÷ prefill time per tokenPlugging in the actual measurements above (6,414 tokens in 768ms, approximately 0.12ms per token), we get 28,672 bytes ÷ 0.12ms ≈ 0.24 GB/s (calculated).

• CPU memory (measured PCIe bandwidth of 13.4 GB/s on this GPU) is more than 50 times faster than the break-even point, making it a clear win.
• A 1GbE-class link (measured effective 104 MB/s in another experiment) is slower than the break-even point. Transferring the KV cache for 6,414 tokens (about 184 MB) takes about 1.8 seconds, which is longer than recomputation (0.77 seconds) (calculated). Loading from slow network storage is a loss.
• The report reaches the same conclusion. On the B200 GPU, a 32Gbps link made loading faster than prefill only when the input exceeded 256K tokens, while 64 and 128Gbps links were faster across all tested lengths (Section 8.7. The model name for this experiment is missing from both the main text and Figure 15 caption—verified by reopening v2 on 2026-10-15. The model list in Section 8.1 is based on H100 experiments). Therefore, the authors recommend choosing between loading and recomputation depending on the situation.

Models and GPUs shift the break-even point. The prefill computation amount is roughly \u201c2 \u00d7 number of parameters\u201d FLOPs, and the KV bytes follow the formula from Part 1. Calculating their ratio (the computation saved by saving 1 byte of KV) for public configurations yields the following (excluding attention calculations for estimation):

The larger the computation per 1 byte of KV, the more advantageous loading becomes. Larger models that reduce KV via GQA have a higher \u201cvalue of loading.\u201d Conversely, the faster the GPU, the faster recomputation becomes, which raises the break-even bandwidth. Even with the same storage, it might be a win on a slow GPU and a loss on a fast GPU.

Hit rate also has a break-even point. A hit saves 527.6ms (576.2 \u2212 48.6), while a miss incurs a storage cost of 22.3ms (597.8 \u2212 575.5). The probability p at which the expected gain becomes zero is 22.3 \u00f7 (22.3 + 527.6) \u2248 4%. This is specific to this experiment and not a universal constant. Plugging in numbers from other public sources raises it to around 20%.

6. Rewriting non-prefix chunks: CacheBlend, CacheGen

Prefix caches are used only when they match from the very beginning . If you concatenate multiple document chunks in a different order every time like in RAG, only the first chunk will hit. If you simply stitch together KV caches created separately for each chunk, the attention between chunks (later chunks looking at earlier chunks) is lost, yielding incorrect answers.

CacheBlend(Yao et al., EuroSys 2025) stitches them together and then recomputes only the tokens where the KV cache deviates the most. Such tokens accounted for about 10\u201315% per layer, and recomputing only 10\u201320% was sufficient. The recomputation is overlapped and hidden behind the loading of the next layer. Results compared to full recomputation: TTFT reduced by 2.2\u20133.3x, throughput increased by 2.8\u20135x, and quality drop (F1/Rouge-L) was below 0.01\u20130.03 (Mistral-7B \u00b7 Yi-34B \u00b7 Llama-70B, dual A40s, measured NVMe 4.8 GB/s). LMCache's MP server enables this method via --engine-type blend .

CacheGen(Liu et al., SIGCOMM 2024) reduces the KV transfer cost itself. It exploits the similarity of KV caches between neighboring tokens by encoding only the differences, quantizing earlier layers densely and later layers sparsely. The KV cache became 3.5\u20134.3x smaller than standard quantization, TTFT was 3.2\u20133.7x lower than the quantization baseline on a 3Gbps link, and accuracy loss was within 2% (Mistral-7B \u00b7 Llama-34B \u00b7 70B, 4x A40s). When bandwidth drops, it lowers the compression level per chunk or switches to sending text for recomputation. This is essentially calculating the break-even point of Section 5 in real time for each chunk.

7. KV transfer in disaggregated prefill-decode architectures

The ability to move KV caches is also used elsewhere. Disaggregated serving, which we will see in Part 4, requires sending the KV cache generated by the prefill GPU to the decode GPU. The disaggregated setup in the LMCache documentation delegates prefill-to-decode transfer to NIXL (part of vLLM's NixlConnector) and couples it with LMCache connectors via vLLM's MultiConnector to further enhance reuse between requests. The report states that under 8K input, 200 output, and NVLink conditions, average TTFT was 1.53\u20131.84x lower and average ITL was 1.12\u20131.66x lower than vLLM's native disaggregated setup (Section 8.5, model name not verified within the section).

8. When is there no benefit?

9. Re-measurement of MP mode on 2026-10-27

Section 4 used the in-process mode now labeled as \u201cLegacy\u201d in the documentation. We re-measured the recommended MP mode (separate lmcache server and LMCacheMPConnector) on the same 8GB-class GPU. Because the model and version differ, do not directly compare these numbers with Section 4; instead, compare the three lines measured on the same day under the same conditions: \u201cvLLM standalone / in-process / MP.\u201d

9-1. Environment and Methodology

• A single 8GB-class laptop GPU (Ada generation), official vLLM 0.32.0 image, LMCache 0.5.6 (official release on 2026-10-14, already included in the image so no separate installation needed), container memory limit 4.5 GiB.
• Model: Qwen3-0.6B (bf16). 28 layers, 8 KV heads, head dimension 128, resulting in 114,688 bytes of KV per token, which is four times that of Qwen2.5-1.5B (28,672 bytes) from Section 4. Conversely, prefill computation per token is about 1.2 GFLOPs, less than half. Expressed using the \u201ccomputation per 1 byte of KV\u201d from the Section 5 table, it is about 10k FLOPs, one-tenth of the 1.5B model (~110k). This is the model least favorable to loading.
• MP mode execution looks like the NVIDIA Dynamo 1.6.0 example (agg_lmcache_mp.sh).

lmcache server --l1-size-gb 0.75 --eviction-policy LRU --port 5555 --http-port 8080
vllm serve Qwen/Qwen3-0.6B --disable-hybrid-kv-cache-manager \
  --kv-transfer-config '{"kv_connector":"LMCacheMPConnector","kv_role":"kv_both","kv_connector_extra_config":{"lmcache.mp.port":5555}}'

• The in-process mode was enabled via LMCacheConnectorV1 and the environment variable LMCACHE_LOCAL_CPU=True, LMCACHE_MAX_LOCAL_CPU_SIZE=0.75, LMCACHE_CHUNK_SIZE=256 .
• Memory budget. The LMCache server used about 0.93 GiB of anonymous memory even before using L1, and the vLLM server used about 2.7 GiB. To fit within the 4.5 GiB limit, L1 was set to 0.75 GB or less. In the first attempt, launching the MP server immediately after shutting down the in-process mode server caused the container to crash due to out-of-memory errors; this was fixed to launch only after confirming all previous servers were down, and re-measured.
• Requests: A short question was appended after a random-word document determined by a seed, queried via max_tokens 2 , and the time to the first token was measured using streaming. The GPU KV capacity was constrained to a small size using --kv-cache-memory-bytes . The same conditions were run three to four times to take the median, and the first pass (including storage) was recorded separately.

9-2. Basic effect: Re-reading document bundles that do not fit entirely in the GPU

GPU KV was constrained to 2,336 tokens, and four documents of about 1,020 tokens each (totaling about 4,080 tokens) were read four times in the same order.

• vLLM standalone fell into the same trap as Section 4. Due to cyclic access, the GPU hit rate was 0.
• Chunk-level penalties were observed. Out of 1,020 tokens, only three 256-token chunks (768 tokens) were loaded, and the remaining ~250 tokens were recomputed. The shorter the document, the larger this fraction is.
• MP mode re-read was about 9ms slower than in-process mode, while the first-pass storage overhead was smaller. This is likely because the KV cache travels between separate processes outside the engine. In MP mode, requests that hit directly on the GPU were also 3\u20134ms slower than vLLM standalone (1,000 tokens: 15.7 vs. 12.0 ms) due to the round-trip overhead of querying the server per request.

9-3. By length: Recomputation vs. loading

GPU KV was set to 0.75 GiB (7,020 tokens), a document was read once and stored, and then vLLM's /reset_prefix_cache(development mode) was used to clear only the GPU cache and re-read it. For LMCache, L1 was cleared per document using the management API. Excluding the first of four runs for each document length, the median was used.

The magnification is smaller than the 1.5B actual measurement in Section 4 (about 19x at 6,414 tokens). This is because this model has fast recalculation and a large KV, so the time saved at the same length is small while the bytes to move are large. However, the pattern of the magnification increasing with length remained the same.

Lessons learned from the first attempt. When running tests by length without clearing L1, most requests did not hit at 2,000 tokens or more. The LMCache logs accumulated "Failed to batched allocate … no enough memory" warnings, and the metric lmcache_mp_l1_allocation_failure_chunks_total{during="l1_store"} increased. Since eviction works by asynchronously clearing 20% when the usage exceeds the threshold (default 0.8), when a large document comes in all at once, it drops the storage instead of waiting for space to become available. Because no errors occur and only hits disappear, L1 should be set generously larger than the largest document expected to arrive at once, and this metric must be monitored.

9-4. Capacity: Partial Hits Instead of a Cliff, and the Backfire of Zipfian Distribution

For cyclic access, 4 documents identical to 9-2 (working set approx. 0.46GB) were used, and for Zipfian distribution, 20 documents of approx. 480 tokens each were read 120 times (exponent 1.1). The GPU KV is 2,336 tokens.

• This time, there was no cliff. Even at 0.25GB, which is smaller than the working set, 44% was loaded. For the in-process mode in Section 4, the hit rate was 0 even when it was just slightly smaller than the working set. It appears that because the MP server's eviction method partially clears at the threshold, some data survived even during circular access.
• In the Zipfian distribution, the MP mode was slower than vLLM alone (17.9 → about 20ms). Frequently read documents remained in the GPU's prefix cache, resulting in identical GPU hits across all three settings (28,208 tokens), and most documents loaded by LMCache were 480 tokens long, so only a single chunk (256 tokens) was retrieved. The lookup and save costs incurred per request outweighed the savings. For workloads where the GPU prefix cache already fits well, it is better not to add LMCache.

9-5. Benefits Exclusive to MP Mode

• The cache persisted even when vLLM was restarted. After reading four documents, only vLLM was shut down and restarted (server left as is, restart took about 30 seconds). Although the GPU cache was empty, all four documents were loaded from the server with 768 tokens, resulting in a time-to-first-token median of 37.5ms, compared to 56.7ms for a newly encountered document in the same fresh vLLM. This is something that cannot be done in the in-process mode because the cache disappears along with the engine process. First requests after deployment, restarts, or sleep modes can be served warmed up.
• Cases where two vLLM instances share a single server were unmeasured. Since one vLLM uses about 2.7GiB and the server uses about 0.93GiB, it was not possible to run two vLLM instances together within the container limit set to half of the node's free memory. The restart test in the first item of 9-5 demonstrates the same principle instead: "another engine process reads the KV stored by the previous process."

9-6. L2 Local Disk Tier and Break-Even Point

L1 was kept small at 0.25GB, and an L2 file system was attached(--l2-adapter '{"type":"fs","base_path":"…","use_odirect":true}', O_DIRECT bypassing the page cache, container's temporary disk).

• Loading from disk was slower than recomputing (55ms vs. 46ms). Although there were L2 hits according to the metrics(lookup_hit_l2_tokens_total 1,024), it went through two stages (disk → L1 → GPU), incurring a fixed delay waiting for prefetching on every request.
• Fitting it into the formula from Section 5 gives the following: For this model, the prefill time per token was about 0.046 ms at 1,000 tokens and about 0.065 ms at 6,000 tokens. The break-even bandwidth, obtained by dividing 114,688 bytes by those times, is approximately 1.8 GB/s (for 6,000 tokens) to 2.5 GB/s (for 1,000 tokens) (calculation). L1 (11.8 GB/s) was more than five times faster than the break-even point, winning as seen in Section 9-3, whereas disk (3.8 GB/s) was only about 1.5 times faster than the break-even point and failed to overcome the fixed delay. If it had been the 1.5B model from Section 4, the break-even point would have been 0.24 GB/s, so it would have won even with the same disk. The smaller the model and the faster the GPU, the lower the value of slower tiers—this conclusion from Section 5 held true.
• Loading long documents (6,000 tokens) from disk could not be measured due to memory limits (unmeasured). There is a possibility that gains would appear as the share of fixed delay decreases.

9-7. Metrics

The Prometheus metrics for the MP server in this setup appeared on the management HTTP port at /metrics (--prometheus-port given via did not respond). lmcache_mp_ There were 48 types of metrics starting with , and the ones to look at first in operations are as follows:

On the vLLM side, vllm:prompt_tokens_by_source_total{source="external_kv_transfer"} is the number of tokens received from LMCache ("external hit tokens" in this section). The documentation states that the management HTTP API has no authentication and is bound only to 127.0.0.1 by default. Since the route list includes /cache/clear, /reconfigure/…, /run_script , exposing this port outside the pod allows anyone to clear the cache or change configurations (default activation status of/run_script unverified).

9-8. Summary

Titi's Word. You don't need to replay a game from scratch every time you want to watch the footage again. You can just play the recording. However, if the recording is in a distant warehouse and traffic is jammed, playing a new game from scratch can sometimes be faster. You have to measure how far away it is and how many times you will rewatch it first.

References

• Liu et al., LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference: https://arxiv.org/abs/2510.09665 (v2, 2025-12-05, Sections 5, 6, 8.2–8.7, 9)
• LMCache Documentation: https://docs.lmcache.ai/ (mp/index.html, mp/l2_storage/supported_storages.html, mp/disaggregated_prefill.html, mp/p2p.html, kv_cache_optimizations/cacheblend.html, accessed October 13, 2026), Releases: https://github.com/LMCache/LMCache/releases
• Yao et al., CacheBlend (EuroSys 2025): https://arxiv.org/abs/2405.16444 (v3, Abstract, Sections 4, 5, 7)
• Liu et al., CacheGen (SIGCOMM 2024): https://arxiv.org/abs/2310.07240 (v6, Sections 5, 7)
• vLLM Disaggregated Prefill & KV Connector: https://docs.vllm.ai/en/latest/features/disagg_prefill/, LMCache Examples: https://docs.vllm.ai/en/latest/examples/disaggregated/lmcache/
• Measurement Citations: Toss ML/AI Infra Preparation, Part 8 (KV Offload Measurement), Part 24 (1GbE Effective Throughput)
• 2026-10-27 MP Mode Measurement Raw Data: Repository docs/experiments/2026-10-27-lmcache-mp/
• NVIDIA Dynamo 1.6.0 Example examples/backends/vllm/launch/agg_lmcache_mp.sh: https://github.com/ai-dynamo/dynamo/blob/v1.6.0/examples/backends/vllm/launch/agg_lmcache_mp.sh