Tag: #DistServe
Kubernetes, DevOps and MLOps, written from things we ran and measured ourselves · 1 posts
NVIDIA Dynamo: Splitting Prefill and Decode, and Sending to Where the KV Exists
Putting prefill and decode on the same GPU causes them to interfere with each other. In practical measurements on an 8GB-class GPU, inserting a single long prompt delayed the next token of a decoding request by 227ms. Di
2026-10-13 · 22 min read #KV 캐시#NVIDIA Dynamo#분리 서빙#KV 인지 라우팅#NIXL