Tag: #실측

← All posts

• KV Offloading and Disaggregated Serving: Reloading Evicted Prefixes from the CPU

  When the GPU prefix cache runs out of space, it evicts the oldest blocks first. vLLM's default CPU offloading (--kv-offloading-size) copies fully used KV blocks to pinned memory and brings them back to the GPU when the s

  2026-10-27 · 12 min read #vLLM#KV 캐시#KV 오프로딩#OffloadingConnector#분리 서빙

• Quantization: Why reducing weights to 8-bit speeds up decoding and the trade-off

  Since a single decode step consists mostly of reading the entire set of weights from memory, making the weights smaller shortens each step. vLLM supports online quantization, which converts bf16 checkpoints to FP8 on the

  2026-10-27 · 13 min read #vLLM#양자화#FP8#AWQ#GPTQ

• Structured Outputs: The Cost of Masking "Illegal Characters" Per Token

  Just before the model selects the next token, structured outputs mask the probability of any grammatically invalid tokens to zero. As a result, even a 0.6B model with almost no conversational ability produced valid JSON

  2026-10-27 · 17 min read #vLLM#구조화 출력#structured outputs#JSON 스키마#xgrammar

• Speculative Decoding: Hit Four Times, Miss and Lose

  Speculative decoding cheaply guesses the next few tokens, and a larger model validates the draft in a single step, accepting the matching ones all at once. In greedy decoding, the output principle remains unchanged. Test

  2026-10-27 · 17 min read #vLLM#추측 디코딩#speculative decoding#n-gram#EAGLE

• Re-measuring continuous batching, chunked prefill, and prefix caching with repeated measurements

  I re-measured vLLM's three core features using a 0.6B model. This time, I repeated each condition three to four times, used the median, and included a baseline with the features completely disabled. Running one request a

  2026-10-27 · 15 min read #vLLM#연속 배칭#chunked prefill#접두사 캐시#cache_salt