Tag: #서빙 옵션
Kubernetes, DevOps and MLOps, written from things we ran and measured ourselves · 1 posts
Measuring vLLM directly with a small model: Load, KV cache capacity, prefix cache, chunk size
I mounted a single 1.7B model onto an 8GB-class GPU, increased concurrent requests, and directly measured where throughput and Time-to-First-Token (TTFT) plateau, whether the KV cache capacity matches the formula, and wh
2026-10-07 · 29 min read #vLLM#부하 테스트#KV 캐시#접두사 캐시#청크 프리필