Hands-On with NVIDIA Dynamo: KV-Aware Routing Got Almost Everything Right, but Disaggregated Serving Was 5x Slower on a Single GPU and Over 20x Slower Across Two Servers

Status: Draft (2026-10-07). All metrics in this post were measured directly by the author. The environment consisted of a single 8GB-class GPU, a 0.6B model, and Dynamo 1.6.0.dev20261006 (development build). The documentation reviewed in Part 6 was v1.5.0, so flags and default values may differ. The test between two servers with different GPUs (1GbE-class network) was measured in Section 4.5, while NVLink and RDMA connections are unmeasured . Writing rule: Empirical data takes priority, and unmeasured items are marked as "unmeasured".

Table of Contents

• 1. Environment and Methodology
• 2. Frontend Overhead
• 3. KV-Aware Routing
  • 3.1 Configuration
  • 3.2 Results
  • 3.3 Silent Failures
  • 3.4 Simulator vs. Empirical Data
  • 3.5 Router in Front of Two Servers
• 4. Prefill/Decode Disaggregation on a Single GPU
  • 4.1 Configuration
  • 4.2 Results
  • 4.3 How to Read These Results
  • 4.4 Why KV Transfer Was Slow
  • 4.5 Disaggregated Serving Across Two Servers (Different GPUs, 1GbE-Class Network)
• 5. Troubleshooting Experience
• 6. Limitations and Unmeasured Items
• 7. Operational Checklist
• References

Part 6 was an overview and summary of Dynamo based on its documentation. To verify whether the documentation's claims hold true in practice, we installed it directly in a single-GPU environment. This post answers four questions:

1. How much latency does the Dynamo frontend add per request?
2. How effectively does KV-aware routing direct requests with repeated prefixes to the same worker compared to round-robin?
3. What happens when the router fails to receive KV events?
4. How does separating prefill and decode into two processes affect the time-to-first-token (TTFT) on a single GPU?
5. How does splitting across two servers with different GPUs (1GbE-class network) affect performance?

Short Answer. Frontend overhead was approximately 13 ms when the server was idle and negligible under load. KV-aware routing directed repeated prefixes to the same worker 53 to 60 times out of 60 trials, compared to 35 to 43 times for round-robin. When the router was configured in event mode but workers failed to emit events, cache hit rates dropped to random levels without generating errors; only a single warning appeared in the logs. On a single GPU, disaggregated serving resulted in TTFT being 4.7 to 7.2 times slower across all conditions. This occurred because both workers shared the same GPU while incurring additional KV transfer overhead, meaning this did not measure the benefits of disaggregation. When splitting across two servers with different GPUs, the TTFT for a 1,500-word input jumped to 1.7 seconds, and for a 3,000-word input to 3.4–3.5 seconds—over 20 times slower than standalone serving (44–172 ms). This increased time matched the KV bytes divided by the link throughput (104 MB/s).

1. Environment and Methodology

• GPU: One 8GB-class GPU. Model: Qwen3-0.6B (28 layers, 8 KV heads, head size 128, bfloat16, 114,688 bytes of KV per token).
• Software: ai-dynamo 1.6.0.dev20261006 (pip dev build), vLLM 0.30.0, NIXL 1.3.2.
• We ran this in a single Python virtual environment without Kubernetes, substituting worker discovery with --discovery-backend file(a shared file). The request plane used TCP defaults.
• Each worker used --gpu-memory-utilization 0.38, a maximum length of 4,096, and eager execution (--enforce-eager). Since two workers shared a single GPU, the available KV slot capacity per worker was 14,816 tokens.
• For the cross-server tests (Section 4.5), we used an additional server equipped with a 24GB consumer-GPU. Because this GPU was shared with other services via time-slicing, we allocated only 18% of the GPU memory ratio (~4.4GB) per worker. Since its driver was for CUDA 12.8, we separately installed a CUDA 12.9 build (Section 5). The TCP throughput between pods on the two servers was 112–113 MB/s.
• Our custom measurement client sent requests one at a time (with concurrency set to 4 only for the disaggregated serving table). Scripts and raw data are available in the repository's docs/experiments/2026-10-07-nvidia-dynamo/ .

2. Frontend Overhead

We compared running the same model using vLLM standalone (solo) versus behind the Dynamo frontend (dyn) with 3 runs each, using an input of 512 tokens and an output of 128 tokens. Values represent the median Time-to-First-Token (TTFT).

At concurrency 1, the median value increased by approximately 13 ms, from 30 ms to 43 ms. At concurrency levels 8 and 32, the difference was indistinguishable, and throughput was identical (1,507 vs. 1,509 tokens/sec at concurrency 32). The overhead of the frontend (HTTP, tokenization, routing decisions) is a fixed cost visible only when the server is idle, becoming masked under heavy load. Because this model is extremely fast, the 13 ms overhead is noticeable; for larger models, its relative impact is expected to be even smaller, though this remains unmeasured .

3. KV-Aware Routing

3.1 Configuration

We launched two workers with the same model and varied only the frontend's --router-mode .

The workload consisted of 12 types of prefixes (approx. 1,500 words, synthetic) sent in shuffled batches of 6, with the trailing 24 words changing each time. This resulted in 72 total requests, with 60 being re-requests where the prefix was sent for the second time or later. A re-request was counted as a cache hit if its TTFT was less than 60% of the initial request (~80 ms). Hits averaged ~15 ms, while misses averaged ~80 ms.

3.2 Results

A prior run with the same configuration yielded consistent trends (Round Robin: 35, 40, 43; Random: 37, 37, 41; Approximate: 60, 60, 60; No Events: 41, 44, 37; Event Mode: 60, 60, 60). The approximate method achieved 60/60 hits across all three runs initially, but scored 53 on a rerun. While this suggests that self-tracking might be slightly less stable than the event-based approach, we cannot confirm this conclusively. The event-based approach achieved 60/60 in all six runs.

3.3 Silent Failures

The fourth row of the table represents the most critical finding in this experiment. When event mode was enabled via --router-mode kv but workers failed to emit KV events, the router's index remained permanently empty. Consequently, all requests evaluated to 0 overlap, causing workers to be selected solely based on load. Hit rates dropped to random levels (39, 44, 41), yet the server responded normally without errors, leaving only a single warning line in the logs. This warning was absent when running with workers that successfully emitted events.

WARN ... KV-event publishing is disabled for a role that requires KV events
diagnostic_code="kv_event_publisher_disabled" ... requirement="cache_aware_routing"

To detect that the hit rate is equivalent to random selection, one must inspect log warnings or separately measure whether the TTFT decreases upon resending the same prefix. If a router is enabled without measuring hit rates, systems can easily run in this degraded state.

3.4 Simulator vs. Empirical Data

Implementing the router's logic in pure Python and running the same workload (selecting the worker with the largest overlapping block from the beginning, breaking ties with the least loaded worker) shows that KV-aware routing achieves 60/60 hits, while lacking event notification yields results identical to round-robin. The directional trends match empirical findings. However, the simulator's round-robin hit rate is 49 out of 60, which is higher than the empirical results (35–43). While the simulator's workers remember processed prefixes indefinitely, real workers have a KV slot capacity of only 14,816 tokens, preventing them from holding all 12 prefix types (~18,000 tokens) and causing older ones to be evicted. Because KV-aware routing distributes prefixes among workers so that each worker holds only 6 types (~9,000 tokens), eviction is avoided. This implies that the benefit of prefix routing includes not only "routing to the same place" but also "reducing the working set size for each worker."

3.5 Router in Front of Two Servers

Placing two workers across two servers with different GPUs (KV slot capacities of 14,816 and 27,088 tokens) and sending the same workload over the inter-server network yielded the following hits out of 60 re-requests: Round Robin (40, 47, 50), Random (47, 40, 52), KV-Aware Approximate (60, 60, 60), and KV-Aware Event Mode (43, 60, 60). KV-aware routing achieved correct routing behavior even in a heterogeneous GPU environment. The reason behind the lower hit count (43) on the first iteration of event mode was not investigated (it is likely that some requests arrived before the event subscription was fully established, but this was unverified). Another limitation is that because initial request speeds differed between servers (44ms vs. 80ms), the hit determination criteria of "less than 60% of the initial request" was less precise in this test.

4. Prefill/Decode Disaggregation on a Single GPU

4.1 Configuration

We deployed dedicated prefill and decode workers as two processes on the same GPU (each using 38% of GPU memory, with KV transfer handled via NIXL), and compared them against a monolithic configuration where a single process handles both phases under the same workload. The output was set to 64 tokens, and values represent the median TTFT across three repetitions.

4.2 Results

Disaggregation was slower across all conditions. Looking at worker KV transfer metrics, concurrency 1 with a 1,500-word input averaged 196 ms per request (119.3 MB per transfer, 608 MB/s throughput), while concurrency 4 with a 3,000-word input averaged 711 to 1,723 ms. For concurrency 1, where the TTFT was about 330 ms slower than the single-process setup, the 196 ms transfer time accounted for a major portion of it. The remaining ~130 ms was not broken down or measured (unmeasured).

).

This is not "disaggregated serving is slow". Since both workers shared a single GPU, the computing resources did not increase, and there was only a single piece of hardware to solve the problem that disaggregation aims to address (long prefills blocking the decodes of other requests). Only the cost of moving the KV cache was added. Cases divided into two servers with different GPUs were measured in Section 4.5. Whether gains occur in environments connected by NVLink or RDMA was not measured in this post (Unmeasured).

Since the KV cache per token is 114,688 bytes, 1,500 tokens amount to approximately 164 MiB. This points in the same direction as the warning in the documentation cited in Part 6 (that moving the KV cache over a slow path can increase TTFT up to about 98 seconds). However, that 98-second figure is for a different model, different input length, and different connection, so its magnitude cannot be directly compared to this experiment. Moving 1,500 tokens at 608 MB/s takes about 280 ms, and at 0.1 GB/s levels like server-to-server TCP, it exceeds 1.5 seconds.

4.4 Why was KV transfer slow?

The cause could not be definitively determined. We tried changing the transfer method, and the results are as follows.

The errors when allowing only IPC were no active messages transport ... cuda_ipc/cuda - no am bcopy and Destination is unreachable . It reads as though NIXL's UCX backend requires a transport capable of sending active messages for initialization, but IPC transport does not provide that. Therefore, the setting to narrow UCX_TLS exclusively to IPC because we wanted to use intra-device transfers cannot be used in this combination. The interface created by UCX under the default configuration was tcp/eth0, tcp/lo, cuda_ipc/cuda .

Because changing the transfer method did not alter the speed, the bottleneck may not be UCX's transport selection. Exactly where the bottleneck lies (request-level serialization, 28 block-level transfer descriptors, copy paths, etc.) was not broken down and measured (Unmeasured).

4.5 Disaggregated Serving Split Across Two Servers (Different GPUs, 1GbE-class Network)

One server was the 8GB-class GPU mentioned above, the other was a 24GB-class consumer GPU (time-shared with another service), and the inter-pod TCP throughput was 113 MB/s. Worker discovery was handled via etcd within one of the pods, and KV transfer used the NIXL default (UCX). The workload was identical (output 64, median Time to First Token across 3 repetitions per slot).

• Splitting across two servers was more than 20 times slower than standalone serving. For concurrency 1, 1,500 words increased from 78ms to 1.75 seconds (40 times slower than the 24GB-class standalone 44ms), and 3,000 words increased from 172ms to 3.5 seconds.
• The increased time is fully accounted for by the network link. The transfer for a single 3,000-word request was 351.75 MB, taking 3,374 ms at 104 MB/s. Since multiplying 114,688 bytes per token by 3,000 tokens yields 344 MB, words and tokens were nearly 1-to-1 for this request. Adding a prefill of 83ms to 351.75 MB ÷ 104 MB/s (approx. 3.4 seconds) results in roughly 3.4 seconds, which matches the measured 3.4~3.5 seconds. For 1,500 words, approx. 176 MB ÷ 104 MB/s yields about 1.7 seconds, matching the measured 1.72~1.75 seconds. In this configuration, the overhead of disaggregated serving could be calculated in advance as "KV bytes ÷ link throughput".
• Which GPU performed the prefill made almost no difference(1,752 vs 1,721, 3,533 vs 3,407 ms). Because the time was bound by the link rather than GPU computation, differences in GPU speed were not visible.
• At concurrency 4, the four transfers had to share the same link. The throughput of each transfer dropped to 26~28 MB/s (summing up to about 112 MB/s for all four), and each transfer took an average of 12.5~13.1 seconds. With only a single link, as concurrency increased, transfers had to queue up, causing the TTFT to increase nearly in proportion to the number of requests.
• This test represents an extreme worst-case scenario for disaggregated serving, using a 1GbE-class link and a 0.6B model (where KV per token is 114,688 bytes and prefill is cheap at 80ms). The trade-offs for cases with much faster links (such as RDMA) or much larger models where prefill is far more expensive can be calculated in the same manner, but were not measured (unmeasured).

5. Incident Report

• Processes were killed due to memory limits. While repeatedly starting two workers and a frontend, memory monitoring triggered a termination while leaving the engine core process running, which then affected the start of the next experiment. We added a script to clean up using pkill -9 EngineCore before each execution.
• The initial router re-test was invalid. The second worker failed to start because it could not acquire the model download lock (.lock), but the script proceeded to run to completion under the assumption that both had started. All three strategies were run against a single worker, rendering the results meaningless. We fixed it so that tests only begin after confirming both workers have successfully secured their KV cache slots, and made sure to pre-download the model once before starting. The invalid logs were left as-is.
• The first attempt to narrow down the UCX transfer method was invalid. The frontend sent requests without discovering the workers, making the measurement meaningless, so we fixed the script and re-ran it.
• Attempted to migrate to a different GPU server, but the driver was incompatible (resolved in three layers). On the server with driver 570 (CUDA 12.8), vLLM 0.30.0 from the default installation (built for CUDA 13) failed to start the engine with Error 804: forward compatibility was attempted on non supported HW . On the server with driver 595, the exact same installation worked out of the box. Resolving this required three steps: (1) First installing the CUDA 12.9 build wheel of vLLM and torch 2.13.0+cu129, and switching NIXL to the CUDA 12 version as well (minor version compatibility allowing the 12.9 runtime to run on the 570 driver). (2) Error 804 still occurred because the container image's ldconfig pointed to the image's CUDA 13-compatible libcuda.so.1 ahead of the host-injected libcuda. After removing that compatibility configuration (/etc/ld.so.conf.d/00-compat-*.conf) inside the pod and re-running ldconfig , the GPUs became visible. (3) During the first sampling after the engine allocated KV cache slots, the kernel compiled by flashinfer for the CUDA 13 runtime crashed with cudaErrorInsufficientDriver (35) , so we disabled the sampler using VLLM_USE_FLASHINFER_SAMPLER=0 . To reuse the same virtual environment installation across different servers, you must first verify whether the driver supports the CUDA build of that installation.
• Narrowing the transfer method caused the engine to fail to start. This refers to the IPC-only configuration from Section 4.4. We modified the script to print out longer error lines to inspect the cause, and preserved the full logs.

6. Limitations and Unmeasured Aspects

• Environments connected via NVLink and RDMA were not measured. Conditions where disaggregated serving becomes beneficial (fast links, large models, long prefills) remain unmeasured . The two-server test (Section 4.5) used a single 1GbE-class link, and the cross-server tests for the router and disaggregated serving involved only two different GPUs. The throughput upper bound when moving the KV cache across servers was constrained by the link (112~113 MB/s).
• The model is small at 0.6B, and the prefixes are synthetic (random words). Results for large models and cache hits on actual documents may differ in terms of Time to First Token gaps.
• Router tests sent requests one by one (concurrency 1). How the router balances load and overlaps when requests coincide was not measured.
• An input size of 1,500 refers to the word count; token counts were not separately tracked. In the cross-server test, a 3,000-word request yielded 351.75 MB per transfer (roughly 3,070 tokens when divided by 114,688 bytes per token), meaning words and tokens were nearly 1-to-1. Meanwhile, the single-GPU test's average of 119.3 MB per transfer would correspond to about 1,040 tokens under the same ratio (just over 70% of 1,500 words), which does not align with this correspondence. We merely suspect that smaller transfers might have been mixed into the average over that 10-second interval, but the root cause was not confirmed (unmeasured). Therefore, it is safer not to read the average values per transfer in Section 4.2 as the exact size of a single request.
• Repetitions were set to three per slot, and standard deviations were not calculated. While trends are solid, it is best not to rely on figures beyond the decimal point.
• Measurements were taken using a development build (1.6.0.dev), so flags and default values may differ from v1.5.0 referenced in Part 6.

7. Operational Checklist

1. If KV-aware routing is enabled, first verify whether workers are emitting KV events . Check frontend logs to ensure there are no kv_event_publisher_disabled warnings.
2. After changing routing configurations, directly measure whether "sending the same prefix again reduces the Time to First Token". It can drop to random-level baselines without error.
3. If prefixes exhibit greater diversity than a single worker's KV cache capacity, it has the additional effect of splitting prefixes across workers. Compare the worker's KV cache capacity (from server logs, GPU KV cache size) against the total sum of prefixes.
4. Evaluate disaggregated serving only after separately measuring the KV transfer path (transfer throughput). In this experiment, transfer times matched calculations closely when links were 1GbE-class (Section 4.5). Multiplying the KV size per token (layers × KV heads × head dimension × 2 × bytes) by the input length yields the bytes transferred per request, and dividing that by the actual transfer throughput gives the additional time added to the Time to First Token.
5. Before each run, verify that no leftover processes (engine cores) or model download locks remain.
6. If NVIDIA driver versions differ across servers, verify first whether the driver supports the CUDA builds of the installed vLLM and torch. If not, it will fail with a CUDA error during the engine startup phase rather than showing up in server logs.

References

• Dynamo Repository and Documentation: https://github.com/ai-dynamo/dynamo
• NIXL: https://github.com/ai-dynamo/nixl
• vLLM's Disaggregated Prefill and NIXL Connector: https://docs.vllm.ai/en/latest/features/disagg_prefill/ , https://docs.vllm.ai/en/latest/features/nixl_connector_usage/
• Qwen3-0.6B configuration (layers, KV heads, head size, data type): https://huggingface.co/Qwen/Qwen3-0.6B/raw/main/config.json
• Part 6 (Distributed Inference and Dynamo, based on documentation) and Part 20 (Measurement Methods) of this series