Tag: #KV캐시
Kubernetes, DevOps and MLOps, written from things we ran and measured ourselves · 2 posts
Hands-On with NVIDIA Dynamo: KV-Aware Routing Got Almost Everything Right, but Disaggregated Serving Was 5x Slower on a Single GPU and Over 20x Slower Across Two Servers
We installed NVIDIA Dynamo on a single-GPU environment without Kubernetes and measured frontend overhead, KV-aware routing hit rates, silent failures when routers fail to receive events, the overhead of prefill/decode di
2026-10-07 · 23 min read #Dynamo#LLM서빙#KV캐시#라우팅#prefill-decode분리Distributed Inference and NVIDIA Dynamo: Parallelism, Prefill/Decode Disaggregation, and KV Cache Movement
This article outlines four parallelism and communication characteristics used when a model does not fit on a single GPU, explains why prefill/decode disaggregation is necessary and how the cost of KV cache transfer is pa
2026-10-07 · 33 min read #LLM서빙#분산추론#Dynamo#병렬화#prefill-decode분리