Tag: #NIXL
Kubernetes, DevOps and MLOps, written from things we ran and measured ourselves · 1 posts
KV Offloading and Disaggregated Serving: Reloading Evicted Prefixes from the CPU
When the GPU prefix cache runs out of space, it evicts the oldest blocks first. vLLM's default CPU offloading (--kv-offloading-size) copies fully used KV blocks to pinned memory and brings them back to the GPU when the s
2026-10-27 · 12 min read #vLLM#KV 캐시#KV 오프로딩#OffloadingConnector#분리 서빙