Tag: #vLLM

← All posts

• Operational Features: Sleep Mode, Startup Time, /metrics, and the Paths That --api-key Ignores

  We tested sleep mode to clear the GPU without shutting down the server, compile caches to reduce startup time, /metrics for monitoring, and authentication firsthand. Step 2 sleep cleared GPU memory from 6,168MiB to 606Mi

  2026-10-27 · 17 min read #vLLM#sleep mode#잠자기 모드#기동 시간#torch.compile

• Parallelization, Multimodal, and Pooling: What You Can and Cannot Benchmark with a Single GPU

  We have grouped the remaining major feature sets of vLLM. Tensor, pipeline, data, and expert parallelism divide models across multiple GPUs. Because we only have a single GPU, these were not benchmarked; instead, we have

  2026-10-27 · 16 min read #vLLM#텐서 병렬#파이프라인 병렬#데이터 병렬#전문가 병렬

• KV Offloading and Disaggregated Serving: Reloading Evicted Prefixes from the CPU

  When the GPU prefix cache runs out of space, it evicts the oldest blocks first. vLLM's default CPU offloading (--kv-offloading-size) copies fully used KV blocks to pinned memory and brings them back to the GPU when the s

  2026-10-27 · 12 min read #vLLM#KV 캐시#KV 오프로딩#OffloadingConnector#분리 서빙

• Tool Calling: The parser turns text into function calls, and strict binds the arguments

  When a model calls a tool, it is actually writing text in a predefined format. vLLM injects a list of tools into the prompt using a chat template, and a parser converts the text written by the model into OpenAI-format to

  2026-10-27 · 15 min read #vLLM#도구 호출#tool calling#function calling#구조화 출력

• Quantization: Why reducing weights to 8-bit speeds up decoding and the trade-off

  Since a single decode step consists mostly of reading the entire set of weights from memory, making the weights smaller shortens each step. vLLM supports online quantization, which converts bf16 checkpoints to FP8 on the

  2026-10-27 · 13 min read #vLLM#양자화#FP8#AWQ#GPTQ

• Structured Outputs: The Cost of Masking "Illegal Characters" Per Token

  Just before the model selects the next token, structured outputs mask the probability of any grammatically invalid tokens to zero. As a result, even a 0.6B model with almost no conversational ability produced valid JSON

  2026-10-27 · 17 min read #vLLM#구조화 출력#structured outputs#JSON 스키마#xgrammar

• vLLM Feature Map: What Exists, When It Benefits, and What Changed in Each Version

  Based on the official documentation and release notes for vLLM 0.32.0 (2026-10-18), sixteen key features are gathered into a single table, detailing what each feature saves, which options enable it, and the conditions un

  2026-10-27 · 15 min read #vLLM#LLM 서빙#기능 지도#PagedAttention#추측 디코딩

• Speculative Decoding: Hit Four Times, Miss and Lose

  Speculative decoding cheaply guesses the next few tokens, and a larger model validates the draft in a single step, accepting the matching ones all at once. In greedy decoding, the output principle remains unchanged. Test

  2026-10-27 · 17 min read #vLLM#추측 디코딩#speculative decoding#n-gram#EAGLE

• Re-measuring continuous batching, chunked prefill, and prefix caching with repeated measurements

  I re-measured vLLM's three core features using a 0.6B model. This time, I repeated each condition three to four times, used the median, and included a baseline with the features completely disabled. Running one request a

  2026-10-27 · 15 min read #vLLM#연속 배칭#chunked prefill#접두사 캐시#cache_salt