Status: Draft (2026-11-14), empirically measured. The numbers in this post were measured directly by the author on 2026-11-13. The environment and requests are the same as in Part 1. Kubernetes v1.34, three GPU servers (one 24GB-class GPU, two 8GB-class laptop GPUs), 1GbE between nodes, the vllm-runtime:1.6.2 image for Dynamo v1.6.2 (vLLM 0.31.0), and the Qwen3-0.6B model. All router arguments are at their default values. Writing convention: Empirical data is the baseline; unmeasured items are marked as "Unmeasured", and unverified items are marked as "Unverified".

Round-Robin and KV-Aware Routing

Table of Contents

• 1. Questions Answered in This Part
• 2. What Changes in the Manifest
• 3. Five Variations
• 4. Results: Round-Robin Matches Baseline, KV Routing Differs
• 5. Time to First Token and Throughput
• 6. Are Results the Same with 3 Workers?
• 7. How Close Did It Get to the Upper Bound?
• 8. Hash Seeds and Approximate Mode
• 9. Conclusion
• References

1. Questions Answered in This Part

The conclusion of Part 1 was as follows: When multiple vLLM instances are placed behind a Service, the prefix cache hit rate is determined by "(number of prefixes held by a single worker) ÷ (total number of prefix types)" and does not increase even when more workers are added. This part sends the same requests to Dynamo to check three things.

1. Is there any difference just by routing through Dynamo (round-robin)?
2. How much do the hit rate and time to first token change when KV-aware routing is enabled?
3. What happens when the two things the documentation warns about—namely, hash seeds and KV events—are removed?

2. What Changes in the Manifest

The baseline used a Deployment and a Service. In Dynamo, those two are replaced by a single DynamoGraphDeployment(DGD). The operator reads this and creates the pods and the Service. Keeping the engine arguments, image, model, and node placement identical, only the following four items were changed.

Two settings are required for KV-aware routing.

# 프런트엔드
env:
  - {name: DYN_ROUTER_MODE, value: kv}

# 워커 인자
- --kv-events-config
- '{"publisher":"zmq","topic":"kv-events","endpoint":"tcp://*:20080","enable_kv_cache_events":true}'

Whenever KV blocks are created and destroyed, workers emit events to this address, and the router inside the frontend receives them to build a map of "which worker holds which blocks." When a request arrives, the router selects the worker whose prefix overlaps the most with the request, while also taking worker load into account.

Here are some facts confirmed during the migration process.

• It started up without etcd, NATS, or Grove. Workers write their own information to the Kubernetes API as a resource named DynamoWorkerMetadata , and the frontend reads it.
• Multiple components of the same type could be deployed. Because resource limits differed per node, we split the type: worker components by node, and the frontend grouped those workers together as workers for a single model.
• Worker metrics are exposed on port 9090. This is a system port injected by the operator. Engine metrics such as vllm:prefix_cache_hits_total were also scraped from here.
• cached_tokens was included in responses without needing to be enabled separately. In a standalone vLLM server, --enable-prompt-tokens-details was a required setting.
• Even after workers became Ready, it took several seconds for the model to appear in the frontend's /v1/models . This list must be verified before applying load.

3. Five Variations

Requests are the same as in Part 1. A shared prefix of about 2,000 tokens was combined with a unique suffix of about 300 tokens per request, and the KV capacity of a single worker was capped at 1GiB (equivalent to about 3.95 requests). Prefix types were 2, 7, 12, and 24; concurrent requests were 4, 8, and 16; and each data point was run 3 times to use the median. Because the seed was identical, all variations received the exact same requests in the exact same order. There were 0 errors across all data points of the five variations.

For one data point in the round-robin variation (2 prefix types, concurrency 4), the first iteration was excluded because the prefix was already in the cache due to a test request sent immediately prior with the same seed. That data point uses the median of the remaining 2 runs.

4. Results: Round-Robin Matches Baseline, KV Routing Differs

Ratio of requests that hit the cache (6 workers, concurrency 8).

The results are similar when viewed as the ratio of input tokens coming from the cache. In this request set, 2,032 out of 2,371 tokens are prefixes, so 0.857 is the maximum possible value.

Round-robin is indistinguishable from the baseline. Simply placing the Dynamo frontend in front does not revive the cache. Distributing requests evenly across workers versus scattering them randomly are equivalent from the perspective of the prefix cache.

KV routing doubled to tripled hit rates in the range where prefixes exceed the capacity of a single worker. The difference is small for 2 prefix types. As noted in Part 1, when prefixes fit entirely within a single worker, routing provides virtually no benefit.

We verified whether the router actually consolidated requests by checking "how many workers a single prefix was sent to." Including nvext.extra_fields in the request's worker_id makes the response include the identifier of the worker that processed it. We counted using that.

(6 workers, concurrency 4, median of 3 runs.) Round-robin sent every prefix to all workers. KV routing sent each prefix to only one worker when there were 2 prefix types, spreading them out as the number of prefixes increased.

5. Time to First Token and Throughput

Time to First Token (TTFT). Values are p50 / p99 per cell (6 workers).

There are three takeaways here.

• The overhead of going through the frontend is a few milliseconds. For 2 prefix types and concurrency 4 where cache conditions are identical, the p50 for round-robin was 41ms versus 37ms for the baseline. At concurrency 8, this widened to 64ms versus 47ms. This is due to the extra step where the frontend tokenizes and passes the request to a worker.
• Round-robin achieved shorter tails despite having the same hit rate. At concurrency 16 and 24 prefix types, the p99 was 774ms, which is half of the baseline's 1,508ms. We suspect this stems from the difference where a Service randomly picks pods per connection—causing moments where requests pile up on a single worker—whereas round-robin distributes them evenly. Since per-worker instantaneous queues were not measured, this cause is speculative.
• KV routing reduced the tail even further. At concurrency 16, the p99 hovers around 0.2 seconds regardless of the number of prefixes. This is because cache misses, which compute all 2,371 tokens and evict other requests on the same GPU, decreased in frequency.

Throughput (requests per second, 6 workers, concurrency 16) also increased accordingly.

Even with the same GPUs, 32% more requests were processed with 24 prefix types. This corresponds to the reduction in time spent recalculating prefixes.

6. Are Results the Same with 3 Workers?

This deployment places one worker per GPU (concurrency 8).

The trend is the same. Throughput increased by 55% with 24 prefix types. However, because there are only 3 workers, the total amount of prefixes that can be stored is halved, and the hit rate is correspondingly lower than the 6-worker deployment.

There was also one point where load exceeded capacity. At 3 workers, concurrency 16, and 24 prefix types, KV routing's TTFT p50 was 279ms and p99 was 962ms. While better than the baseline (563ms, 3,060ms), it did not drop below 0.2 seconds like the other points. Routing only reduces waste; it does not manufacture insufficient compute resources.

7. How Close Did It Get to the Upper Bound?

The theoretical upper bound calculated in Part 1 was "(number of workers) × (number of prefixes held by one) ÷ (total prefix types)." This value assumes prefixes are distributed across workers without overlap.

Default KV routing achieved between half and two-thirds of the upper boundin regions where the cache was constrained. While two to three times better than random distribution, it falls short of fully utilizing capacity without waste.

The reason is shown in the table in Section 4. At 24 prefix types, a single prefix went to an average of 5 workers. Since the router considers worker load in addition to overlap, if the overlapping worker is busy, it sends the request to another worker. That worker computes and caches the prefix from scratch, which evicts other prefixes in the process. The tighter the cache, the more this eviction triggers a cascade.

The Dynamo documentation includes router arguments to adjust the weight between overlap and load. This article measured only the default values; whether tweaking those arguments can bring it closer to the upper bound remains unmeasured.

8. Hash Seeds and Approximate Mode

The documentation and examples state to give the workers a PYTHONHASHSEED=0 . The reason given is that because vLLM's block hash goes through Python's hash function, if the seed differs per process, the same prefix will result in different hashes across workers. I retested without the seed (6 workers, concurrency 8, cache token ratio).

Removing the seed yielded the same results. The differences across the four points were within 0.03 and their directions varied. This magnitude is indistinguishable from the fluctuation between iterations. At least in this setup and configuration, routing did not break without the seed. The reason remains unverified. I could not confirm in the code whether the runtime sets the seed itself or if there is a way for the router not to rely on the worker's hash. Since this is a setting required by the documentation and carries no cost to include, I do not recommend omitting it just because the results were the same.

Approximate mode falls in the middle. In this mode, workers do not send events, and the router guesses that "that worker probably still has this prefix" based on the history of requests it has sent. While there was little difference with KV events for 7 prefixes (0.638 vs 0.702), for 24 prefixes, the score dropped to 0.245, cutting the difference with round robin (0.149) in half. This is likely because when the cache is tight and blocks are frequently evicted, the router's guesses diverge from reality. This should be understood as a fallback option to be used in environments where opening event ports is difficult.

9. Conclusion

• Simply placing Dynamo in front does not revive the prefix cache. The hit rate of round robin was the same as vLLM behind a Service.
• KV-aware routing proves its worth in ranges where the total volume of prefixes exceeds the capacity of a single worker. With 6 workers, the request-level hit rate went from 56% to 83% for 7 prefixes, 32% to 68% for 12 prefixes, and 18% to 43% for 24 prefixes, while the TTFT p99 at concurrency 16 dropped from 1.3–1.5 seconds to the 0.2-second range.
• For a service where all prefixes fit into a single machine, there is almost nothing to gain. The hit rate difference was 2–4%p for 2 prefixes. The costs incurred in that case are a few milliseconds for an extra frontend layer and the burden of installation and permissions seen in Part 2.
• By default, it yields half to two-thirds of the theoretical upper limit. Adjusting router parameters was not measured.
• Removing the hash seed produced identical results (reason unverified). Approximate mode showed diminishing returns as the cache became tighter.

The limitations of this measurement are the same as in Part 1. Prefixes were sampled evenly, request lengths were identical, and the model is small. With a larger model, the computation for a single missed request will take much longer, meaning the same hit rate difference will manifest as a much larger time difference, but that is not what this article measured. Whether routers see the same map when multiple frontends are deployed was also not measured.

References

• NVIDIA Dynamo v1.6.2 Documentation, KV Cache Routing and Router Configuration.
• Manifests, load testing tools, metric logs, and median summaries can be found in the repository docs/experiments/2026-11-13-dynamo-adoption/ .
• The principles of KV events and routers are covered in An Anatomy of LLM KV Caches Part 4.