Status: Draft (2026-10-13), partially measured. Papers, official documentation, and the NVIDIA blog were directly opened and verified on 2026-10-13, and the documentation is based on Dynamo v1.5.1 (2026-10-07). Performance figures released by NVIDIA are marked as "Vendor Announcement." Measurements were re-cited from "Toss ML/AI Infra Prep" Part 14 (8GB-class GPU, vLLM) and Part 24 (Dynamo 1.6.0 dev build, 8GB-class and 24GB-class GPUs, 0.6B model) on the same site. The disaggregated serving benefits in NVLink- and RDMA-connected environments are unmeasured. Writing rules: Measured data is the default; unmeasured items are marked as "Unmeasured," and items that could not be verified are marked as "Unverified."

NVIDIA Dynamo: Splitting Prefill and Decode

Table of Contents

• 1. Prefill and Decode Interfere with Each Other on the Same GPU
• 2. Disaggregated Serving: Split and Move the KV
• 3. Architecture of NVIDIA Dynamo
• 4. Why Throughput Increases
• 5. Dynamo Measured Directly
• 6. Relationship Between Dynamo and LMCache
• 7. When There Is No Benefit
• 8. Series Summary
• References

1. Prefill and Decode Interfere with Each Other on the Same GPU

As mentioned in Part 1, prefill is bound by compute, while decode is bound by memory bandwidth. Problems arise when the two share a single GPU. Continuous batching inserts the prefill of a new request between decode steps, but steps with long prefills take a long time, during which other requests in the batch cannot receive their next token.

Our Measurement. On an 8GB-class GPU with default vLLM settings (max input per step 2,048 tokens), inserting a 3,500-token input alongside a request currently undergoing decoding resulted in a maximum token interval of 227ms in that segment (227, 226, and 226ms across 3 trials). The usual interval is about 15ms. Chunked prefill from Part 2 could reduce this value down to 69ms, but at the cost of slightly delaying the first token of long inputs. Chunking only distributes the interference, it does not eliminate it.

Two papers that systematically addressed disaggregated serving for the first time point out this problem as follows.

• DistServe(Zhong et al., OSDI 2024): Placing both on one GPU means (1) the two phases slow each other down, worse with longer prompts, and (2) the two phases must use the same resources and the same parallelism settings, forcing GPUs to be over-provisioned based on whichever is stricter between TTFT and token interval targets. They evaluate chunked prefill as merely trading off TTFT against token intervals.
• Splitwise(Patel et al., ISCA 2024): Analyzing real cloud traces, they showed that the prompt phase is compute-intensive while the token phase is memory-bandwidth and capacity-intensive. In BLOOM-176B, processing a 1,500-token prompt took the same time as generating 6 output tokens, and the token phase showed almost no increase in latency even when GPU power was lowered from 700W to 350W. They conclude that when characteristics differ this much, it is better to use different hardware or different configurations.

2. Disaggregated Serving: Split and Move the KV

The method is simple. Maintain separate GPUs dedicated to prefill and GPUs dedicated to decode. Once the prefill GPU generates the prompt's KV, it sends that KV to the decode GPU, and the decode GPU continues token generation using the received KV.

• Since long prefills do not interrupt the decode GPU, token intervals are regular. While maintaining interval targets, larger decode batches can be accommodated.
• Different parallelism and different numbers of GPUs can be given to the two phases. Prefill is tuned toward partitioning compute, while decode is tuned toward accommodating large batches. the two phasesdifferent parallelism and different numbers of GPUs
• The price paid is KV transfer. As formulated in Part 1, KVs are large and must be moved before the first token.

Results from DistServe. Measured on A100 80GB (NVLink within nodes, 25Gbps between nodes) using OPT-13B, 66B, and 175B for the maximum request rate (goodput) satisfying latency targets by 90% or more.

Compared to another baseline (DeepSpeed-MII), it was up to 7.4x, and the abstract's statement "up to 7.4x request rate or 12.6x stricter latency target" combines these two tables. Synchronization examples are also intuitive. Putting a 13B model on a single A100 achieved about 1.6 requests/sec, whereas prefill alone achieved 5.6, decode alone achieved 10 requests/sec, and splitting into 2 prefill + 1 decode GPUs yielded 3.3 requests/sec per GPU, a 2.1x increase.

What about KV transfer? The paper calculates that a single 512-token request for OPT-66B amounts to about 1.13GB of KV, requiring 11.3GB/s at 10 requests/sec. Therefore, it was designed to place prefill and decode within the same node (NVLink) , resulting in KV transfer accounting for less than 0.1% of total latency in OPT-175B.

Results from Splitwise. KVs were sent asynchronously layer by layer to overlap with the next layer's computation (InfiniBand). Unoverlapped transfers took about 8ms on A100 and about 5ms on H100, which was less than 7% of prompt computation time and added 0.8% to total latency. Under equal power conditions, an A100-only split configuration achieved 2.15x the throughput of the baseline (A100) (Figure 18a in the paper). The representative figures in the abstract ("1.4x at 20% cost reduction," "2.35x at equal cost/power") could not be matched with pairs from the main experiments, so only main text values are noted here.

3. Architecture of NVIDIA Dynamo

Dynamo is a serving framework that ties together multiple GPUs and multiple nodes on top of engines like vLLM, SGLang, and TensorRT-LLM. It does not replace engines; instead, it handles how to partition multiple engines and where to send requests. The architecture confirmed in the documentation (v1.5.1) is as follows.

• Frontend: OpenAI-compatible API, tokenization.
• KV-Aware Router: Determines which worker to send requests to. It calculates costs for each worker and chooses the cheapest one. The cost formula in v1.5.1 is "prefill blocks that need to be newly computed on that worker (excluding blocks already held) × weight + decode block load + running request count × weight." Workers that already possess the KV for the same prefix come out cheaper because their prefill share is reduced. However, load is also factored in to prevent excessive concentration in a single place. The documentation explains that increasing the weight given to prefix overlap reduces TTFT, while decreasing it spreads the load evenly and reduces token intervals.
• How workers know what they possess: The default method is for workers to send an event whenever blocks are saved or deleted so the router can build an index; an approximation mode (default TTL 120 secs) that remembers "recently sent requests" without events is also available.
• Disaggregated Serving: A separate router selects the prefill worker, and after the prefill worker generates the KV, the decode worker directly fetches that KV via NIXL .
• NIXL: A library wrapping transfers between GPU memory, CPU memory, files, and object stores into a unified interface. Actual transfers are handled by plugins (UCX by default, GPUDirect Storage, POSIX, Mooncake, libfabric, object stores, etc.). It utilizes NVLink, RDMA such as InfiniBand/RoCE, PCIe, and TCP depending on the situation.
• Planner: Dynamically adjusts the number of prefill and decode workers based on load. It supports modes such as throughput targets (queue depth, KV utilization; default) and latency targets (TTFT, token interval SLA), with methods ranging from predicting incoming request counts, input/output lengths, and cache hit rates, to applying regression to engine step-by-step metrics. In Kubernetes, an operator aligns these counts.
• KV Block Manager (KVBM): This was a tier cascading KVs down from GPU (G1) → pinned host memory (G2) → local disk (G3) → object store (G4). It was marked as deprecated in the v1.5.0 release notes and is scheduled for removal in v1.7.0. Users are advised to migrate host and disk tiers to engine-native offloading (or integrations like LMCache).

4. Why Throughput Increases

NVIDIA's announced performance figures are as follows. All are Vendor Announcementsand have not been independently reproduced.

• Introduction blog (2025-03-18): Running DeepSeek-R1 671B (FP4, 32K input / 8K output) on GB200 NVL72 via disaggregated serving achieved up to 30x throughput, and Llama 70B (FP8, 3K input / 50 output) on Hopper achieved over 2x. Figure footnotes note "Expected performance, subject to change."
• Design document for v0.9.0: Disaggregated serving increases per-node GPU throughput by +30% on the Llama 70B family (FP8, H100, 3K input/150 output) and over 2x across 2 nodes. KV-aware routing sends 100,000 real requests (average 4K input/800 output) to two H100 nodes, reducing TTFT by 3x and average latency by 2x compared to random routing. KVBM CPU offloading improves TTFT by 2.2x to 12x on Qwen3-8B (H100, multi-turn dialogue, average 20K input) compared to using only the GPU prefix cache.

5. Dynamo Measured Directly

We measured Dynamo (1.6.0 dev build, vLLM 0.30.0, NIXL 1.3.2) in a separate series using Qwen3-0.6B (114,688 bytes of KV per token).

KV-Aware Routing. We spun up two workers and sent 12 types of prefixes of about 1,500 words each, mixed in 6-fold repetition (60 repeat requests). When a repeat request went to a worker that already had the matching prefix KV, the time to first token was about 15ms; otherwise, it was about 80ms.

• Enabling the router resulted in nearly 100% hits.
• Beware of silent failures. If Event Mode was enabled but workers failed to send events, the router's index remained empty, dropping hit rates to random levels while leaving only a single warning in the logs without any errors. After enabling routing, you must directly measure whether resending the same prefix actually reduces the time to first token.
• Round-robin did not perform any better than random. Since a single worker's KV capacity (14,816 tokens) could not fit all 12 prefix types (~18,000 tokens), KV-aware routing partitioned the prefixes into 6 types per worker so each could store them fully. The benefit of routing comes from combining "sending to the same place" with "shrinking each worker's working set."
• The overhead of passing through the frontend was about 13ms when idle, and was indistinguishable at concurrency levels of 8 and 32.

Disaggregated Serving. The results were the opposite. Everything was slower than single serving.

• Splitting a single card does not increase compute resources, it only adds KV transfer overhead. This is a setup where there is no GPU to solve the problem disaggregation is meant to address.
• When split across two servers, the increased time matched KV bytes ÷ link throughput. Transferring a single 3,000-word request took 351.75MB, 3,374ms, and 104MB/s; adding 83ms of prefill to about 3.4 seconds of transfer matched our measurements. It was nearly identical regardless of which GPU handled the prefill, because the time was bound by the link rather than computation.

This does not mean "disaggregated serving is slow," but rather **"splitting over a slow link is slow."** This table explains why papers presuppose NVLink or InfiniBand.

6. Relationship Between Dynamo and LMCache

The two are not competitors; they operate at different layers. LMCache is a cache layer that stores and loads KV alongside a single engine, while Dynamo is a framework that orchestrates, splits, and dispatches across multiple engines. The integration points identified in the documentation are as follows.

• Aggregated Configuration (One worker handles both prefill and decode): Dynamo's vLLM worker offloads and loads KV to/from the node's LMCacheMPConnector(role kv_both) via lmcache server . Because workers and LMCache servers attach via CUDA IPC, Kubernetes requires hostIPC: true . The v1.5.1 Kubernetes guide covers only this aggregated configuration, specifies verified version combinations separately, and warns that mismatched LMCache versions between the server and worker will result in communication protocol mismatches.
• Disaggregated Configuration: The v1.3.0 documentation provides an example where the prefill side wraps both the LMCache connector and the NIXL connector together. Prefill-to-decode transfer is handled by NIXL, while LMCache handles reuse between requests—a division of labor. The LMCache documentation's disaggregated configuration follows the same pattern (Part 3, Section 7). We could not find official procedures for using LMCache with disaggregated serving in v1.5.1 (unverified).
• Relationship with KVBM: Dynamo's offloading overview document introduces LMCache, SGLang's HiCache, and FlexKV as alternatives, advising not to overlap them. KVBM has been deprecated since v1.5.0.
• Router Visibility Scope: The v1.5.1 router support matrix states that vLLM's native offloading, SGLang's HiCache, and TensorRT-LLM host/disk hits are factored into its scoring (0.75 for host hits, 0.25 for disk), while the LMCache row is absent. Therefore, we could not confirm from the documentation whether the router incorporates KVs offloaded to LMCache into its scoring (unverified).

7. When There Is No Benefit

The cost-benefit trade-off of disaggregated serving can be estimated using an equation similar to the one in Part 3: Time added to first token ≈ KV per token × input tokens ÷ link throughput(layered transfers allow some portion to be hidden overlapping with computation). Calculating the time to transfer KV for a 4,096-token input yields the following (transfer time only, excluding computation).

Link speeds are roughly 1GbE-class for our measured effective 104MB/s, 100Gb/s for a theoretical 12.5GB/s, and NVLink-class for the 600GB/s between A100s reported in the DistServe paper.

If this cost exceeds what can be saved (decoding slowed down by interference, or over-provisioned GPUs), it results in a net loss. Consequently, disaggregation yields no benefit or is detrimental in the following scenarios:

• Slow Links. The first column in the table above illustrates this case, where actual measurements showed it was over 20x slower.
• Short Prompts. If the prefill is short, the time injected into decoding is also short, leaving little interference to eliminate. Meanwhile, the fixed costs of transfer and routing remain.
• Low Load. When few requests are decoding concurrently, there is no competing target. Since disaggregation splits the GPU into two, it only increases idle GPU capacity when traffic is light.
• Single GPU. Splitting does not increase compute resources (measured 4.7x to 7.2x slower).
• Small Models. Prefill is already cheap (83ms for 0.6B prefill), making KV transfer overhead relatively large.

Dynamo documentation also notes that there is no universal threshold. Because boundaries shift depending on the model, GPU, and input length distribution, the initial release (v0.1.0) utilized remote prefill "only when the prefill length minus prefix hits exceeded the threshold and the remote prefill queue was short." In v1.5.1, conditional splitting based on router overhead has entered the experimental stage. It is safest to decide whether to disaggregate after first measuring your link's KV transfer throughput and evaluating the equation above with that value.

KV-aware routing is a different story. Because it moves requests toward the KV rather than moving the KV , there is no link overhead. Under workloads where identical prefixes recur frequently, it is relatively unlikely to incur losses. However, you must tune load weights to prevent overloading a single worker and verify that events are actually arriving.

8. Series Summary

A single question runs through all four parts: "Which is cheaper: recomputing this KV, or fetching it from somewhere else?" The answer is determined by the model's KV size per token, the GPU's compute speed, the speed of the path the KV travels, and how frequently the same prefix recurs. All four are measurable values.

A TT Note. If you put someone practicing serves and someone practicing rallies on the same table, they both lose out waiting for each other to clear the ball. Splitting the tables makes it better. But what if you have to carry baskets of balls over from the next building? In that case, you are better off just playing together. Measure how fast the ball basket arrives before you split.

References

• Zhong et al., DistServe: Disaggregating Prefill and Decoding for Goodput-optimized LLM Serving (OSDI 2024): https://arxiv.org/abs/2401.09670 (v3, Sections 1, 2.3, 3.3, 6, Table 1, Figures 8, 9, 10)
• Patel et al., Splitwise: Efficient Generative LLM Inference Using Phase Splitting (ISCA 2024): https://arxiv.org/abs/2311.18677 (v2, Sections 3, 4.C, 6, Figure 18), Microsoft Research page: https://www.microsoft.com/en-us/research/publication/splitwise-efficient-generative-llm-inference-using-phase-splitting/
• NVIDIA Dynamo Repository & Releases: https://github.com/ai-dynamo/dynamo (v1.5.1, 2026-10-07), Documentation: https://docs.nvidia.com/dynamo/ (Architecture, router concepts/configuration, disaggregated serving, planner, KVBM configuration, KV offloading/LMCache MP deployment, router offloading support matrix, accessed 2026-10-13), Early disaggregated serving documentation: https://github.com/ai-dynamo/dynamo/blob/v0.1.0/docs/disagg_serving.md
• NIXL: https://github.com/ai-dynamo/nixl (README, docs/nixl.md)
• NVIDIA Introductory Blog (Vendor Announcement): https://developer.nvidia.com/blog/introducing-nvidia-dynamo-a-low-latency-distributed-inference-framework-for-scaling-reasoning-ai-models/ (2025-03-18), Design Document v0.9.0 (Vendor Announcement): https://docs.nvidia.com/dynamo/v-0-9-0/design-docs/overall-architecture
• LMCache Dynamo Integration Documentation: https://docs.lmcache.ai/production/dynamo_coordination.html
• Measurement Citations: Toss ML/AI Infra Preparation, Part 14 (Chunks and Interference), Part 24 (Dynamo Measurements)