Tag: #접두 캐시
Kubernetes, DevOps and MLOps, written from things we ran and measured ourselves · 2 posts
Empirical measurement of round-robin vs. KV-aware routing when sending the same requests to Dynamo
We sent the requests measured in Part 1 back to Dynamo in the same order. When the frontend was configured for round-robin, the hit rate was identical to vLLM behind a Service. Simply passing through Dynamo does not revi
2026-11-13 · 16 min read #NVIDIA Dynamo#vLLM#KV 인지 라우팅#접두 캐시#KV 캐시How Well Does Prefix Cache Hit behind Multiple vLLM Instances in a Service?
When you group multiple vLLM pods behind a single Kubernetes Service, requests go to random pods. Because prefix caches are isolated per pod, requests with the same prefix that land on different pods are recomputed from
2026-11-13 · 11 min read #NVIDIA Dynamo#vLLM#접두 캐시#KV 캐시#쿠버네티스