Distributed Inference and NVIDIA Dynamo: Parallelism, Prefill/Decode Disaggregation, and KV Cache Movement

Status: Draft (2025-10-05, banner updated on 2025-10-07). This text is based on official documentation, and the figures in this article still reflect documentation standards. Direct measurements of Dynamo installed on a single-GPU environment (frontend cost, KV-aware routing, single-GPU prefill/decode disaggregation) are covered separately in Part 24. Disaggregated serving between two servers with different GPUs (over a 1GbE-class network) is also measured in Section 4.5 of Part 24, while NVLink and RDMA environments remain unmeasured . Please read "Verified Versions and Dates" and "What Was Not Verified" at the end of the article first. Writing Rule: Measured data is the default, and unmeasured items are marked as "Unmeasured".

Table of Contents

• Summary
• 1. Four Ways to Split a Model
• 2. Prefill/Decode Disaggregation: Why Do It and What Is the Cost
• 3. Through Which Path Does the KV Cache Move
• 4. Architecture of Dynamo v1.5.0
• 5. Positioning Differences from llm-d, vLLM, and SGLang
• 6. Where Disaggregation Is Beneficial Versus Where It Hurts
• 7. Hands-on Experiment
• External Evidence: Findings from Papers and Official Materials
• 8. Potential Interview Questions
• 9. Common Misconceptions
• References (Actually Opened and Checked)
• Verified Versions and Dates, and Unverified Items

Summary

Because large models do not fit on a single GPU, their weights must be distributed across multiple devices. Since communication volume and latency vary significantly depending on the splitting method (tensor, pipeline, expert, or data parallelism), high-bandwidth approaches tend to be used within a node, while low-communication approaches are preferred across nodes. In addition, because prefill (prompt processing) and decode (token generation) have different resource characteristics, splitting their pools prevents them from interfering with each other, but requires moving the KV cache instead. NVIDIA Dynamo is the layer that orchestrates this disaggregation, KV-aware routing, and autoscaling on top of the engine.

What You Can Explain in an Interview After Reading This Article

1. What tensor, pipeline, expert, and data parallelism each split, what communication they trigger, and why choices differ within a node versus across nodes.
2. The motivation and cost of prefill/decode disaggregation explained through KV cache size calculations, along with the criteria distinguishing workloads that benefit from it versus those that suffer.
3. How Dynamo v1.5.0 components process a single request, and how it differs from llm-d, vLLM, and SGLang.

The author's standpoint. I am experimenting in a standard Ethernet (1–2.5GbE class) environment and have no experience with RDMA, InfiniBand, or NVLink. This article summarizes findings from reading official documentation and papers, and the only direct measurements are from a single LMCache experiment.

1. Four Ways to Split a Model

Communication volume is estimated using the formula from Narayanan et al. (2021). PP sends b·s·h elements (the product of batch size, sequence length, and hidden dimension) per microbatch between adjacent segments. In training, TP sends and receives 8·b·s·h·(t−1)/t per device per layer because all-reduce occurs twice in both forward and backward passes. Since inference lacks a backward pass, keeping only the forward pass share yields 4·b·s·h·(t−1)/t (derived by the author).

Illustrative example (not measured). When prefilling a single 8,000-token prompt on Qwen3-32B (hidden dim 5120, 64 layers) with TP=8 and bf16 computation, about 287MB per device per layer, and roughly 18GB total across all 64 layers, is exchanged via all-reduce. For the same request, data crossing a single PP boundary is about 82MB.

Therefore, papers suggest using TP up to the number of GPUs within a server and extending beyond that with PP (Takeaway #1 in a training context). vLLM documentation also recommends setting TP to the number of GPUs per node and PP to the number of nodes, noting that PP is better for GPUs lacking NVLink (such as L40S). Bandwidth differences are also visible in specifications. NVIDIA lists 4th-gen NVLink at 900GB/s per GPU and 5th-gen at 1,800GB/s (bidirectional sum), whereas a single ConnectX-7 NIC is 400Gb/s (approx. 50GB/s per direction). The H800 cluster in the DeepSeek-V3 report featured 160GB/s NVLink versus 50GB/s InfiniBand.

Because EP sends tokens to the GPU hosting the expert and receives results back, an all-to-all operation occurs at every layer. DeepSeek-V3 restricted routing so that tokens visit at most 4 nodes, reducing InfiniBand traffic, and set its deployment units to 32 GPUs for prefill (EP32) and 320 GPUs for decode (EP320). In vLLM, turning on --enable-expert-parallel makes the EP size TP × DP , and it guides using deepep_high_throughputfor prefill and deepep_low_latency for decode as communication backends.

 [노드 A: TP=4]                          [노드 B: TP=4]
 GPU0 GPU1 GPU2 GPU3 -- PP 경계 --> GPU4 GPU5 GPU6 GPU7
  └ NVLink all-reduce ┘  (활성화 텐서,    └ NVLink all-reduce ┘
    층마다, 지연 민감       점대점, RDMA)      층마다, 지연 민감

2. Prefill/Decode Disaggregation: Why Do It and What Is the Cost

Motivation. Dynamo documentation describes prefill as compute-bound and decode as memory-bound, allowing a larger TP for decode and a smaller TP for prefill. SGLang documentation explains that mixing them in a single engine causes new prefills to interrupt ongoing decodes, delaying token generation. vLLM documentation cites the ability to independently tune TTFT (Time to First Token) and ITL (Inter-Token Latency) and control tail ITL as reasons, while clearly noting that disaggregation does not increase throughput. On the research side, DistServe reports request rates 2.0–4.6x higher than vLLM (1.6–7.4x higher than DeepSpeed-MII) for chatbots and meeting SLOs up to 12.6x more strictly for summarization (these are maximums across different applications and comparison targets, not values obtained simultaneously in a single configuration). Splitwise claims 1.4x throughput with 20% lower cost, or 2.35x at the same power, compared to a traditional cluster (cluster-scale gains are simulator results, not measurements). Both values were derived under paper conditions using 70B to 175B class models and fast interconnects (intra-node NVLink, 200/400Gbps InfiniBand). The figures in the body were verified directly in "External Evidence" below.

Cost: The KV cache must be moved. KV size per token is 2(K와 V) × 층 수 × KV 헤드 수 × 헤드 차원 × 바이트 수 . Calculated with Qwen3-32B's config.json(64 layers, 8 KV heads, head dimension 128, bf16), it is 256KiB per token, or about 2.1GB for 8,000 tokens.

Author's measurements. This is an experiment running Qwen2.5-1.5B with vLLM+LMCache on an 8GB-class laptop GPU. Loading 4,864 tokens of KV cache (0.13GB) per document from CPU memory to GPU took about 8–9ms, while recomputing the same 5,000 tokens took about 560ms. Since the KV size per token for this model is 28KiB, 4,864 tokens become 0.13GiB, which matches the 0.1299 in the logs. However, this is movement from CPU to GPU within a single node rather than network transfer, with concurrency 1, a 1.5B model, a synthetic document, and a single measurement. The takeaway is simply that "moving can be cheaper than recomputing," and for this to hold across a network segment, the bandwidth must support it as shown in the table above.

The Dynamo documentation set notes that moving KV via TCP yields a TTFT of about 98 seconds, whereas RDMA yields 200–500ms. The nature of these two figures differs. The 98-second figure was measured on Llama-3.1-8B with an 8,000-token input on AWS p5.48xlarge when the NIXL plugin on the EFA pod silently fell back to UCX's slow path (with the documentation advising to specify the NIXL backend as LIBFABRIC as a solution), while 200–500ms is a value the documentation describes as a "specification-based expectation." The additional KV transfer overhead actually measured on EFA under the same conditions was 37ms. Therefore, "TCP takes 98 seconds and RDMA takes 0.5 seconds" should not be read as a general comparison (see "External Evidence" below).

3. Through Which Path Does the KV Cache Move

The transfer library is NIXL (NVIDIA Inference Xfer Library). It abstracts memory types (HBM, DRAM, files, object storage) and selects backends via plugins. Backends mentioned in documentation include UCX and GPUDirect Storage, with Dynamo docs noting it goes through UCX or libfabric, and SGLang supporting Mooncake and NIXL. Within a node, CUDA IPC is used for NVLink, and between nodes, RDMA is used.

The direction is a "pull" model where decode pulls from prefill. The llm-d documentation states this, and vLLM's NixlConnector holds onto prefill-side KV blocks with a default 30-second lease. There are four operational points to keep in mind. First, NCCL_IB_HCA variables like NCCL do not apply to NixlConnector, and you must use UCX_TLS, UCX_NET_DEVICES instead. Second, according to Dynamo documentation, NVLink is blocked even between different pods on the same node, forcing transfers through the NIC. Third, in Kubernetes, the /dev/shm default value (64MB) is insufficient for NIXL initialization, so it is recommended to raise sharedMemory to 16Gi. Fourth, vLLM documentation marks kv_role="kv_both" as deprecated and uses kv_producer/kv_consumer to be used, but the Dynamo 1.5.0 documentation example still uses kv_both .

4. Structure of Dynamo v1.5.0

Dynamo does not replace the engine; instead, it coordinates multiple nodes on top of vLLM, SGLang, and TensorRT-LLM. The runtime is written in Rust, and extensions are in Python.

flowchart LR
  C[클라이언트] -->|HTTP 8000| F[Frontend<br/>토큰화·검증]
  F --> R[Router / PrefillRouter]
  R -->|1 prefill 요청| P[Prefill Worker]
  P -->|2 disaggregated_params| R
  R -->|3 decode 요청+메타데이터| D[Decode Worker]
  P -.->|4 KV 전송 NIXL| D
  D -->|5 토큰 스트림| F
  PL[Planner] -.->|복제본 수 조정| P
  PL -.-> D
  DS[(Discovery: K8s 또는 etcd)] -.-> F

• Frontend: An OpenAI-compatible HTTP server (port 8000) responsible for applying chat templates, tokenization, validation, and detokenization.
• Router: Selects the cheapest worker by considering both workers whose prefixes are already cached and the current load. The cost is prefill_load_scale × max(0, 활성 prefill 블록 + 신규 프롬프트 블록 − 겹침 크레딧) + 잠재 decode 블록 + 활성 요청 비용 . Workers publish KV creation and deletion events to update the index, and if events are not received, prediction mode is used with --no-router-kv-events . The documentation refers to this as "batch decision," and even a worker with a hot cache will lose if its load is high.
• Worker: The engine process. In a disaggregated deployment, roles are separated into --disaggregation-mode prefill|decode .
• Planner: Adjusts the number of replicas targeting TTFT and ITL SLAs. The targets are throughput(default) and latency, load, sla , and throughput-based (default 180-second period) and load-based (default 5-second period) policies can be used together. Because workers are terminated during scale-down without waiting for in-flight requests, it is recommended to set a lower bound for min_endpoint .
• Communication Plane: Discovery (Kubernetes uses DynamoWorkerMetadata and EndpointSlice, bare metal uses etcd), requests (TCP default, NATS optional), events (ZMQ default, NATS optional).
• Changes and Deprecations: KV Block Manager (KVBM) is deprecated in v1.5.0 with removal targeted for v1.6.0 (migration to the engine's KV offloading is recommended). AIConfigurator was renamed to AISimulate, and Go Endpoint Picker was replaced with a Rust version.

Kubernetes. The Dynamo Operator reconciles CRDs. The core is DynamoGraphDeployment(DGD, a set of inference graphs consisting of a frontend and workers), along with DynamoComponentDeployment, DynamoGraphDeploymentRequest(DGDR, which profiles models, hardware, and SLAs to generate DGDs), DynamoGraphDeploymentScalingAdapter(integrated with HPA/KEDA), and DynamoModel . In v1.5.0, nvidia.com/v1beta1 becomes the stored version, so spec.components uses type: frontend|worker|prefill|decode in the list. For multi-node workers, the Operator returns an error unless Grove (default) or LeaderWorkerSet is present. The following is a shortened version of the documentation example (Qwen3-32B) and has not been applied in practice.

apiVersion: nvidia.com/v1beta1
kind: DynamoGraphDeployment
spec:
  components:
  - {name: Frontend, type: frontend}
  - name: VllmPrefillWorker
    type: prefill
    sharedMemorySize: 16Gi
    # args: --model Qwen/Qwen3-32B --disaggregation-mode prefill
    #       --kv-transfer-config '{"kv_connector":"NixlConnector",...}'
  - name: VllmDecodeWorker
    type: decode        # args 는 같고 --disaggregation-mode decode

5. Position Comparison with llm-d, vLLM, and SGLang

The three projects are not mutually exclusive. The Dynamo 1.5.0 release notes include an llm-d batch gateway example, and both use vLLM and SGLang as model servers. We could not verify which one is faster. llm-d's "up to 70% throughput" (B200, AWS) and Dynamo's "2x TTFT" (Baseten) not only have different conditions but also measure different targets. The 70% figure represents throughput with prefill/decode disaggregation (GPT-OSS, 1,024 input and output tokens), while the 2x figure represents TTFT with KV-aware routing rather than disaggregation (Qwen3-Coder 480B, approx. 50k input tokens, compared to random routing), so the two cannot be placed side by side to determine superiority.

6. Where Disaggregation is Beneficial vs. Harmful

Based on Dynamo documentation, benefitsoccur when long prompts make prefill expensive, high concurrency makes decode a bottleneck, separate scaling of the two pools is desired, or large models require different parallelization strategies across the two stages. llm-d cites long contexts with a 10:1 input-to-output ratio as a recommended use case. Drawbacksoccur with small models, short prompts, low concurrency, and a lack of fast KV transfer networks, where an aggregated deployment is simpler and often faster. The documentation also warns that when decode KV capacity is the bottleneck, scaling up prefill replicas can reduce overall capacity.

Dynamo's countermeasure is conditional disaggregation (experimental) . If the effective input length—calculated by subtracting overlapping decode worker KVs—is less than 2,048 tokens by default and the ratio relative to the original length is less than 0.7, the decode worker handles the prefill directly without passing through a prefill worker. This targets cases with heavy prefix reuse, such as multi-turn conversations, aligning with the same design concern behind vLLM's bidirectional KV transfer (where prefill pulls previous-turn KVs held by decode).

7. Trying It Out Yourself

Without RDMA and multi-GPU nodes, disaggregated serving cannot be reproduced, but the following scope is feasible:

1. KV Size Calculation (No environment changes). Calculate KV per token for model config.json .

   python3 -c "import json,sys;c=json.load(open(sys.argv[1]));h=c.get('head_dim',c['hidden_size']//c['num_attention_heads']);print(2*c['num_hidden_layers']*c['num_key_value_heads']*h*2,'bytes/token(2바이트 가정)')" config.json
   

2. Measure KV Loading Cost. Run vLLM and LMCache in a temporary pod as in the experiment in Section 2, comparing the time to load KV from CPU memory versus recomputing it. Delete the pod when finished. If node memory is shared with other workloads, headroom must be verified before increasing limits.
3. Run NIXL over TCP (Unverified). Spin up prefill and decode instances on two single-GPU servers and restrict UCX_TLS to TCP families to see if TTFT increases like the 1GbE line in the table above. This was not executed while writing this article, and proper operation is not guaranteed. Because GPUs may be occupied by other workloads, verification is required prior to execution. According to external measurements, even at 100Gb/s TCP, inter-GPU transfer varies depending on the implementation from 0.55GB/s (UCX) to about 4.8GB/s (UCCL), so the theoretical line rate of the 1GbE line may actually be slower (see "External Evidence" below). (Added 2026-10-07: Direct measurements between two servers with different GPUs were taken in Section 4.5 of Part 24. With default UCX settings (TCP), the first token for a 1,500-word input took 1.7 seconds, and for a 3,000-word input took 3.4–3.5 seconds, which is over 20 times slower than standalone serving (44–172ms). A variant restricting UCX_TLS to the TCP family within a single GPU showed the same speed as the default settings.)
4. Directly measured node-to-node TCP throughput (minimum actual measurements required for calculation). Across three different pairs of nodes (pod network, iperf3 for 10 seconds), single-stream throughput was 894–902 Mbit/s and 4-stream throughput was 901–912 Mbit/s, both in the 1GbE class (approx. 112 MB/s). Moving an 8,000-token KV (approx. 917 MB) for a 1.7B model at 114,688 bytes per token over this network takes about 8 seconds, whereas recomputing the same input on an 8GB-class GPU takes about 1 second (Part 21). In other words, disaggregated serving is disadvantageous for small models and 1GbE(transfer is 8 times slower than recomputation), confirming the statement in Section 6 through calculation. End-to-end TTFT directly connecting the two instances was measured in Section 4.5 of Part 24. A 3,000-word request for a 0.6B model (same KV size of 114,688 bytes per token) transferred 351.75 MB per transfer at 104 MB/s in 3.4 seconds, and the first token took 3.4–3.5 seconds, matching the calculation (standalone 83–172ms).

External Evidence: Verified via Papers and Official Materials

Because disaggregated serving cannot be reproduced without datacenter-grade equipment, all numerical claims in this article were verified against primary sources along with their conditions. External figures differ in conditions from our environment (small model, single GPU, 1GbE), so they were used only to judge direction rather than for direct comparison.

Vendor figures (NVIDIA, AWS, Baseten, llm-d) were transcribed only for conditions stated in the main text. The original blogs for two llm-d README entries and the communication volume formula for Megatron-LM could not be accessed, so figures were omitted.

8. Potential Interview Questions

1. If deploying a 70B model across two 8-GPU nodes, how would you configure parallelism? Since NVLink is available within a node, we set TP=8, and since inter-node communication is lower, we set PP=2. However, since PP introduces bubbles, if the model fits on a single node, it is better to place TP replicas on each node and scale out using DP.
2. Why and when do you disaggregate prefill and decode? The motivation is to separate compute-bound and memory-bound resource characteristics, eliminate mutual interference, and independently control TTFT and ITL. Since KV transfer and GPU splitting are the costs, aggregation is better for short prompts, low concurrency, and slow networks.
3. How long does it take to transfer the KV of an 8K token prompt over 100GbE? With KV per token (256KiB for Qwen3-32B) totaling about 2.1GB and a line rate of 12.5GB/s, it theoretically takes 0.17 seconds. Since reality takes longer, the answer is to determine the need for RDMA by comparing this with the TTFT target.
4. What are the advantages and risks of KV-aware routing? Routing to a worker with the prefix skips prefill calculation and reduces TTFT, but concentrating traffic on a worker with the cache creates a hotspot. Dynamo puts cache credit and active load into a single cost function, and advises checking the effect by comparing random|round-robin and kv under the same load.
5. How does Planner differ from HPA? It uses TTFT and ITL SLAs rather than CPU or request count, uses signals per prefill/decode (queue depth, KV utilization), and predicts traffic. Because in-flight requests may fail during scale-down, a minimum replica lower bound is set.

9. Common Misconceptions

• "Disaggregation is always faster." vLLM documentation notes that it does not increase throughput. The purpose is latency separation and tail latency control.
• "NCCL_IB_HCA also applies to KV transfer." NixlConnector uses UCX, so configure UCX_NET_DEVICES .
• "KVBM is the core of Dynamo." It is scheduled for deprecation in v1.5.0. It remains in the README feature table, so please refer to the release notes.
• "Increasing TP always makes it faster." All-reduce occurs at every layer, so crossing nodes causes communication to dominate. Dynamo is also a coordination layer above engines, not a new engine.

References (Actually Accessed)

• Dynamo v1.5.0 Release Notes: https://github.com/ai-dynamo/dynamo/releases/tag/v1.5.0
• Dynamo Repository README and Documentation (release/1.5.0 docs/fern/pages: Architecture, disaggregated serving, KV-aware routing, router concepts, conditional disaggregation, planner, Kubernetes disaggregated serving·DGD·RDMA·installation, API reference, glossary): https://github.com/ai-dynamo/dynamo , https://docs.nvidia.com/dynamo/
• NIXL: https://github.com/ai-dynamo/nixl
• vLLM: https://docs.vllm.ai/en/latest/features/disagg_prefill/ , https://docs.vllm.ai/en/latest/features/nixl_connector_usage/ , https://docs.vllm.ai/en/latest/serving/parallelism_scaling/ , https://docs.vllm.ai/en/latest/serving/expert_parallel_deployment/
• SGLang: https://docs.sglang.ai/advanced_features/pd_disaggregation.html
• llm-d: https://github.com/llm-d/llm-d , https://llm-d.ai/docs/architecture , https://llm-d.ai/docs/well-lit-paths/foundations/pd-disaggregation
• Papers: https://arxiv.org/abs/2104.04473 , https://arxiv.org/abs/2412.19437 , https://arxiv.org/abs/2401.09670 , https://arxiv.org/abs/2311.18677
• NVLink Specifications: https://www.nvidia.com/en-us/data-center/nvlink/ , Qwen3-32B config: https://huggingface.co/Qwen/Qwen3-32B/raw/main/config.json

Verified Versions, Dates, and Unverified Items

Verification date is 2026-10-05. Checked via GitHub Releases API for Dynamo v1.5.0 (2026-09-21, bundled engines SGLang v0.5.18, TensorRT-LLM v1.3.0rc25, vLLM v0.28.0), NIXL v1.5.0 (2026-09-30), llm-d v0.10.0 (2026-09-29), vLLM latest v0.30.0, SGLang latest v0.5.21, and the documentation site latest is also v1.5.0.

Unverified items: (1) Why some Dynamo documentation still uses v1alpha1 and kv_both examples. (2) Performance superiority among llm-d, Dynamo, and SGLang (comparison is difficult due to differing measurement targets). (3) Section 7 Experiment 3 (testing TTFT by running NIXL over TCP between two servers) was executed on 2026-10-07 in Section 4.5 of Part 24 (only on 1GbE-class links, not RDMA). (4) Some original blogs for two llm-d README entries and the communication volume formula in the Megatron-LM paper could not be accessed and were omitted. Conditions for 98 seconds and 200~500ms (now verified), and DistServe/Splitwise body figures (now verified) are in "External Evidence".