Status: Draft (2026-11-14), measured. The numbers in this post were measured directly by the author on 2026-11-13. The environment consists of Kubernetes v1.34, three GPU servers (one 24GB-class GPU and two 8GB-class laptop GPUs), and 1GbE between nodes. The engine is vLLM 0.31.0 included in the Dynamo runtime image vllm-runtime:1.6.2 , and the model is Qwen3-0.6B. Because these values were measured with a small model and a small cache, absolute time cannot be directly transferred to larger models; what does transfer is the manner in which hit rates are determined. Writing rule: Empirical data is the default. Unmeasured items are marked as "Unmeasured", and unverified items as "Unconfirmed".

vLLM Prefix Cache behind a Service

Table of Contents

• 1. Why Measure the Baseline First?
• 2. What and How Was Measured?
• 3. Results: Hit Rates Collide as Prefixes Increase
• 4. How Much Do Hits and Misses Differ in Time to First Token?
• 5. What This Baseline Implies
• References

1. Why Measure the Baseline First?

This series documents the empirical process of introducing NVIDIA Dynamo into a Kubernetes environment that previously used only vLLM. One of the standout features promoted by Dynamo is KV-aware routing, which routes incoming requests to workers that already hold the matching KV cache for the request's prefix.

To quantify its value, we first need to know "how much cache the current approach wastes." The current approach refers to placing a single Service in front of multiple vLLM pods. This post measures only that configuration; Dynamo will be covered starting from the next part.

Let us briefly touch upon one background concept. vLLM's prefix cache reuses the KV cache of a request's prefix without recalculating it if a previous request shared the same prefix. In services that prepend system prompts or long documents, this significantly reduces the Time to First Token (TTFT). However, this cache exists separately per process . Pod B does not know about the KV cache calculated by Pod A.

2. What and How Was Measured?

We intentionally constrained the KV capacity to 1GiB. Since a single request is 2,371 tokens, a single worker holds enough KV cache for only about 3.95 requests. This setup was designed to create conditions where exceeding 4-prefix types means they cannot all fit into a single worker at once. It mimics the cache pressure on smaller hardware that real services experience when using large models and long contexts.

There are two deployment configurations.

• 6 workers: 4 workers sharing a 24GB-class GPU via time-slicing, and 1 worker on each of two 8GB-class GPUs.
• 3 workers: 1 worker per GPU.

The load generator was run on a separate server without a GPU, sending requests to the Service's ClusterIP. Three rules were followed to ensure a fair measurement.

1. Establish a new connection for every request. The Service selects pods on a per-connection basis. If the client reuses connections, it remains pinned to the same pod, making the baseline appear artificially better.
2. Fix the request sequence using a seed. The exact same requests in the exact same order are sent to the Dynamo variants in later sections.
3. Exclude warmup from aggregation. Requests are sent once per prefix before measurement, and those requests are not counted.

Hits were verified in two ways. One is via the response's usage.prompt_tokens_details.cached_tokens , and the other is the difference in measurement intervals scraped from each vllm:prefix_cache_hits_total ÷ vllm:prefix_cache_queries_total . Both values matched to the third decimal place in 23 out of 24 data points, and the remaining point differed by only 0.001. For example, with 2 prefix types and concurrency of 4, both values were 0.814.

There was one pitfall. A standalone vLLM server does not include --enable-prompt-tokens-details in the response unless cached_tokens is provided. Measuring without this flag makes all requests appear to be misses.

3. Results: Hit Rates Collide as Prefixes Increase

Ratio of requests that hit the cache (concurrency 8, median of 3 runs).

The pattern remains the same at concurrency levels 4 and 16. For 6 workers, hit rates were 52.5% and 52.0% for 7 prefixes, 28.7% and 32.1% for 12 prefixes, and 16.0% and 16.7% for 24 prefixes.

Two things are worth noting.

First, the hit rate matches a simple formula. When the Service evenly distributes requests, each worker receives requests mixed with all prefixes. Because workers evict the least recently used blocks first, a single worker's cache retains roughly "the 3.95 most recently seen prefixes." The probability that the next request's prefix falls within that set is 3.95 ÷ number of prefixes. The last column in the table shows this value, which matches the measurements within a 1-3 percentage point margin.

Second, increasing the number of workers does not improve the hit rate. The results for 6 workers and 3 workers are nearly identical. This happens because random load balancing causes independent behavior on each worker. The total KV capacity is doubled, but the hit rate remains unchanged. Combining the capacity of 6 workers can hold about 23 prefix types, meaning that under a 24-prefix load, capacity that could have preserved most of the cache was reduced to effectively utilizing only 16%.

The reason hit rates do not reach 100% even with 2 prefixes can also be explained mathematically. With 6 workers and 2 prefix types, there are 12 (worker, prefix) combinations. The 2 warmup requests fill 2 of them, and the remaining 10 combinations miss once when first encountered. Measured values across all three runs yielded 190 hits out of 200. For 3 workers, 4 misses occurred out of 6 combinations minus 2, resulting in 196 hits, which was also identical across all three runs.

4. How Much Do Hits and Misses Differ in Time to First Token?

We analyzed TTFT at concurrency 4 by dividing requests into hits and misses (6 workers, combined across 3 runs).

Hits take about 40ms, and misses take about 105-113ms. The latency of individual requests does not change as the number of prefix types increases; only the ratio between the two groups changes. Consequently, the median of the overall TTFT shifts toward the miss side as the number of prefixes increases.

(6 workers, median of 3 runs.) Tail latency grows longer at concurrency 16. With 2 prefix types, the p99 was 325ms, but from 7 prefixes onwards it reached 1.3 to 1.5 seconds. This occurs because missed requests must compute all 2,371 tokens from scratch, and that computation delays other requests running on the same GPU.

Throughput also decreased accordingly. With 6 workers at concurrency 16, throughput was 18.9 requests per second for 2 prefix types and 13.3 for 24 prefix types. For 3 workers, it dropped from 25.1 to 11.7 requests per second.

Even though the 3-worker deployment did not time-slice GPUs, it suffered from longer tails at concurrency 16 (p99 of 3,060ms for 24 prefix types) because the same load was handled by half the number of workers. Conversely, throughput was higher with 3 workers for 2 prefix types, as time-slicing a single 24GB-class GPU across four workers does not increase compute resources. Comparing absolute times between the two deployments is not the goal here; the key takeaway is that hit rates follow the same formula regardless of deployment.

5. What This Baseline Implies

In summary, the prefix cache hit rate for vLLM behind a Service is determined by the following formula:

적중률 ≈ min(1, 워커 한 대가 담는 접두 수 ÷ 자주 쓰이는 접두의 종류 수)

The number of workers is absent from this formula. Therefore, we can deduce the following:

• If all prefixes fit within a single worker, changing the routing yields almost no benefit. With 2 prefix types, the only loss is a single initial miss per combination.
• Once prefixes begin to exceed the capacity of a single worker, random load balancing duplicates capacity across the number of workers. Multiple workers independently compute and store the same prefixes, crowding out different prefixes in the process.
• Routing requests to workers by prefix reduces the number of prefix types each worker handles. Theoretically, the upper bound increases to "number of workers × prefixes held by a single worker ÷ total number of prefix types." With 6 workers, this calculates to achieving high hit rates even with 24 prefix types.

This final point is what KV-aware routing promises. How closely it approaches that upper bound and what the cost of introducing an extra frontend layer is will be measured by resending identical requests in the same order.

Limitations of this measurement should also be noted:

• Prefixes were sampled uniformly. In real services, certain prefixes are used much more frequently, meaning random load balancing would yield higher hit rates than this. Measurements reflecting popularity distributions remain unmeasured.
• Request lengths are uniform. When lengths are mixed, "prefixes held by a single worker" is no longer a fixed value.
• KV capacity was fixed at 1GiB. Providing ample capacity avoids pressure even with the same number of prefixes. The key metric to observe is not the number of prefixes, but the ratio of total prefix volume to single-worker capacity.
• GPU types are mixed. Different worker speeds may have widened the TTFT distribution.

References

• vLLM Documentation, Automatic Prefix Caching.
• Measurement tools and raw data summaries are located in the repository at docs/experiments/2026-11-13-dynamo-adoption/ .
• The principles of KV cache and bytes-per-token calculations are covered in LLM KV Cache Anatomy Part 1, and Dynamo's architecture and previous measurements are in Part 4.