Tag: #양자화
Kubernetes, DevOps and MLOps, written from things we ran and measured ourselves · 2 posts
Quantization: Why reducing weights to 8-bit speeds up decoding and the trade-off
Since a single decode step consists mostly of reading the entire set of weights from memory, making the weights smaller shortens each step. vLLM supports online quantization, which converts bf16 checkpoints to FP8 on the
2026-10-27 · 13 min read #vLLM#양자화#FP8#AWQ#GPTQvLLM Feature Map: What Exists, When It Benefits, and What Changed in Each Version
Based on the official documentation and release notes for vLLM 0.32.0 (2026-10-18), sixteen key features are gathered into a single table, detailing what each feature saves, which options enable it, and the conditions un
2026-10-27 · 15 min read #vLLM#LLM 서빙#기능 지도#PagedAttention#추측 디코딩