Tag: #Marlin
Kubernetes, DevOps and MLOps, written from things we ran and measured ourselves · 1 posts
Quantization: Why reducing weights to 8-bit speeds up decoding and the trade-off
Since a single decode step consists mostly of reading the entire set of weights from memory, making the weights smaller shortens each step. vLLM supports online quantization, which converts bf16 checkpoints to FP8 on the
2026-10-27 · 13 min read #vLLM#양자화#FP8#AWQ#GPTQ