Status: Draft (2026-11-14), Measured. The numbers in this post were measured directly by the author on 2026-11-13. The environment and requests are the same as in Part 1. The setup includes Kubernetes v1.34, a single 24GB-class GPU server and an 8GB-class laptop GPU server, 1GbE between nodes, no RDMA, the Dynamo v1.6.2 vllm-runtime:1.6.2 image (vLLM 0.31.0), and the Qwen3-0.6B model. We clarify upfront that this condition is unfavorable for separate serving. Writing rules: Measured data is the default, unmeasured items are marked as 「Unmeasured」, and unverified items are marked as 「Unverified」.

Re-measuring Separate Serving in a Small Cluster

Table of Contents

• 1. What Separate Serving Aims to Do
• 2. Configuration
• 3. Separation Within the Same Server
• 4. Separation Between Servers: Slow, and Memory Filled Up First
• 5. When Is It Beneficial Then?
• 6. Conclusion
• References

1. What Separate Serving Aims to Do

Processing a single request is divided into two tasks: reading the entire prompt to create the KV cache (prefill), and generating tokens one by one while looking at that KV cache (decode). Prefill performs a large computation all at once, whereas decode repeatedly performs small computations for a long time. When both run on the same GPU, a long prefill halts token generation for other requests.

Separate serving assigns these two tasks to different workers. The prefill worker creates the KV, transfers it to the decode worker, and the decode worker generates the tokens. The cost is the time required to transfer the KV. Dynamo uses a library called NIXL for this transfer, and the documentation explains it under the premise of RDMA. It states that it also runs over TCP, but is much slower.

This part aims to answer three questions:

1. Can a separate serving graph be deployed in Kubernetes without Grove?
2. What is the cost of separation within the same server in an environment without RDMA?
3. What is the cost of separation crossing 1GbE?

2. Configuration

We placed type: prefill and type: decode in the DGD components and specified the roles in the worker arguments.

# prefill 워커
- --disaggregation-mode
- prefill
- --kv-transfer-config
- '{"kv_connector":"NixlConnector","kv_role":"kv_producer","kv_buffer_device":"cpu"}'

# decode 워커
- --disaggregation-mode
- decode
- --kv-transfer-config
- '{"kv_connector":"NixlConnector","kv_role":"kv_consumer","kv_buffer_device":"cpu"}'

# 둘 다
env:
  - {name: UCX_TLS, value: "tcp,self"}
  - {name: UCX_RNDV_THRESH, value: "inf"}

Since there was no RDMA, the transfer was fixed to TCP. kv_buffer_device was set to cpu following the comments in the Dynamo examples. It notes that attempting TCP over GPU buffers causes crashes within UCX. Under this setting, the KV passes through host memory. A warning stating "UCX CUDA support was not found" was also left in the worker logs.

There are two deployments:

The frontend uses the KV routing mode as in Part 3. The engine arguments and requests are also the same. Since a single request averages 2,371 tokens and the KV for a single token is 114,688 bytes, the KV for a single request is approximately 272MB .

It deployed without Grove. When applying DGD, the operator created separate Deployments for the frontend, prefill, and decode, and all five pods reached the Ready state within two minutes. The frontend sent requests to the prefill worker first and then passed them to the decode worker. Grove is turned off in the chart defaults and was not needed for a graph of this size.

3. Separation Within the Same Server

The medians are based on prefix types of 2, 7, 12, and 24, concurrency levels of 4 and 8, and 3 runs per data point. All 24 measurements had zero errors. The comparison baseline is the KV routing configuration (6 workers) from Part 3.

The two configurations do not have identical resources. The KV routing setup used 4 workers on the same 24GB-class GPU and utilized 2 additional workers on another server. The separation setup used only 4 workers (2 prefill, 2 decode) on that single GPU. Therefore, the difference in throughput is partially influenced by the share of those 2 extra workers. On the other hand, the difference in time to first token cannot be explained by that.

This is evident by looking at the most favorable cell. With 2 prefix types and concurrency 4, the cache token ratio was 0.857, meaning the prefill worker hardly had to recompute the prefixes. Yet, it took 286ms to get the first token. In the KV routing setup, the same cell was 34ms. Even if there is a cache hit on the prefill worker, the decode worker does not possess that KV and must receive the entire 272MB. The increased duration of approximately 250ms accounts for that cost. Transferring 272MB in 0.25 seconds equates to just over 1GB per second, which is a plausible speed for copying via TCP through host memory within the same server. Since transfers were not measured in isolation, this breakdown is an estimation.

As concurrency increases, transfers overlap and cause further delays. At concurrency 8, the median was 0.46–0.9s, and p99 was 1–1.8s.

Hit rates also dropped. The cache token ratio was 0.40–0.42 for 12 prefix types and around 0.20 for 24 types, falling short of KV routing (0.57–0.66 and 0.36–0.43, respectively). This is because there are only two prefill workers holding the prefix cache, meaning the total cache capacity is one-third of what it is with 6 workers.

To summarize, under this condition, separation within the same server resulted in a first-token latency 8 to 12 times slower and a 37 to 48% reduction in throughput. Because there was only a single GPU, computational resources did not increase, and since it was a small model where prefill finishes within 100ms, the issue of "long prefill blocking decode" was minor to begin with. This configuration merely added transfer costs where nothing was to be gained.

4. Separation Between Servers: Slow, and Memory Filled Up First

1GbE practically carries 110–117MB per second. Calculating for transferring 272MB yields an expected duration of 2.3–2.5 seconds.

We were unable to complete measurements for this deployment because the prefill worker was OOMKilled after just a few requests.

The time to first token for the six successful requests was 1.6–2.7s, which matches the size calculated from the link speed. With only six samples, we cannot describe the distribution. Compared to the 34ms of KV routing, this is roughly 50 times slower.

The memory aspect is a more useful finding. The memory limit for the prefill worker was 3,584Mi. This was the limit that successfully ran all measurements in Part 3 using the same image and engine arguments. kv_buffer_device: cpu In this setup, because the KV to be transferred is loaded into host memory, an additional 272MB is consumed per request. At concurrency 4, this exceeds 1GB. Given that even at concurrency 1 the limit was breached after four requests, it appears that buffers finished with transfers were not reclaimed immediately. Since worker memory usage was not tracked over time, the root cause is unverified.

This issue did not occur in the same-server deployment, where the worker limit was 10Gi. We did not re-measure the cross-server deployment with an increased limit because that server shared other workloads, preventing us from allocating a generous limit.

There is one key takeaway here: switching to separate serving requires recalculating worker memory limits. While the situation may differ in configurations using GPU buffers over RDMA (unmeasured), configurations using TCP and host buffers require adding "number of concurrent requests × KV size per request" plus some headroom on top of the existing limit.

5. When Is It Beneficial Then?

This post did not measure the conditions under which separate serving wins. What we measured were two conditions where it loses. However, the measurements did reveal what factors dictate the trade-off.

분리 서빙이 치르는 값 ≈ 요청 하나의 KV 바이트 ÷ 전송 처리량
분리 서빙이 얻는 값 ≈ 긴 prefill 이 같은 GPU 의 decode 를 막던 시간

• The cost incurredcan be calculated in advance. It is obtained by multiplying the KV bytes per token by the input length and dividing by the link throughput. For the model and requests in this post, this was about 0.25 seconds within the same server and about 2 seconds over 1GbE. The cost increases for models with larger KV caches and longer inputs.
• The benefits gainedmaterialize when models are large, inputs are long enough for prefill to take seconds, and prefill and decode can be placed on separate GPUs. The prefill for a 0.6B model takes under 100ms, so there was nothing to be gained.

This equation explains why the documentation premises its setup on RDMA. Transfer throughput must reach tens of GB/s for the cost incurred to become smaller than the prefill time. The cost-benefit balance in environments with RDMA or NVLink is unmeasured.

6. Conclusion

• The separate serving graph deployed successfully using just DGD without Grove. All that is needed are the component's type and the two worker arguments.
• Splitting via TCP within the same server increased the time to first token from 34–106ms to 286–896ms and reduced throughput by 37–48%. This occurs because even cache-hit requests require transferring the entire KV.
• Crossing 1GbE resulted in a first-token latency of 1.6–2.7s (6 successful requests). A prefill worker with a 3,584Mi memory limit was OOMKilled within 9 seconds at concurrency 4, and after four requests at concurrency 1.
• In separate serving using TCP and host buffers, "number of concurrent requests × KV size per request" must be added to the worker's memory limit.
• For small models, a single GPU, and networks without RDMA, there is no reason to enable disaggregated serving. On the same hardware, Part 3's approach of using only KV routing was better across all metrics.

To summarize the entire series line by line: vLLM behind a Service drops the cache if a prefix exceeds the capacity of a single instance (Part 1). To introduce Dynamo, you first need to check the operator's webhook scope and permissions (Part 2). KV routing multiplies the hit rate by two to three times in that section (Part 3). Disaggregated serving is for when you have a fast network and a large model (this part).

References

• NVIDIA Dynamo v1.6.2 documentation and examples on disaggregated serving.
• vLLM documentation, Disaggregated Prefilling and NixlConnector.
• Manifests(d3-disagg.yaml), benchmark records, and worker logs are located in the repository's docs/experiments/2026-11-13-dynamo-adoption/ .
• Records measuring KV transfer volume and throughput separately between two servers without Kubernetes can be found in Trying out NVIDIA Dynamo directly.