Quantization: Why reducing weights to 8-bit speeds up decoding and the trade-off

Status: Draft (2026-10-27), measured for FP8, GPTQ, and AWQ; GGUF and bitsandbytes unmeasured. Measured on 2026-10-27 using a single 8GB-class laptop GPU (Ada generation), vLLM 0.31.0, and Qwen3-Reranker-0.6B. Initial measurements were skipped because pre-quantized checkpoints (AWQ/GPTQ) needed to be downloaded anew, and were measured in Section 3-1 after approval. Actual measurements of the 8-bit KV cache capacity are available in Episode 14 of "Toss ML/AI Infra Preparation". Added on 2026-10-27. Upon approval, the vLLM 0.32.0 image, Qwen3-0.6B, Qwen3-0.6B-GPTQ-Int8, and Qwen3-4B-AWQ (all Apache-2.0, safetensors) were pulled to add Section 3-1. The previous 0.31.0 figures were left as is. Writing conventions: Measured data is the default. Unmeasured items are marked as "Unmeasured", and unverified items are marked as "Unverified".

Table of Contents

• 1. Why it gets faster
• 2. vLLM quantization map
• 3. Measuring it yourself: Online FP8
  • Method
  • Results
• 3-1. 2026-10-27 Remeasurement: Chat models and pre-quantized checkpoints
• 4. 8-bit KV cache
• 5. Takeaways for operations
• 6. Unverified items
• References

1. Why it gets faster

During decoding, to generate a single token per request, the entire set of weights is read once from GPU memory. For a 0.6B model, 1.2GB in bf16 is transferred back and forth at every step. Since computation is relatively small, step time is largely determined by "weight bytes ÷ memory bandwidth." Reducing weights to 8-bit halves the bytes to read, thereby shortening the step.

Two implications follow from this:

• The larger the batch, the smaller the gain. When there are more requests in a single step, weight reads remain the same while KV reads and computations increase, reducing the share of step time taken up by weights.
• If even activations can be computed in 8-bit (W8A8), computation also becomes faster. This is possible on GPUs with FP8 tensor cores (Ada, Hopper, and later). Approaches that reduce only the weights while keeping computation at 16-bit (W4A16, W8A16) only reduce memory reads.

2. vLLM quantization map

The hardware table in the official documentation shows that FP8 W8A8 runs only on Ada and Hopper or newer, while AWQ and GPTQ run on most architectures since Turing. The numbers reported in papers vary by conditions. AWQ reported being over 3x faster than Hugging Face FP16 implementations on desktop and mobile GPUs with 4-bit weights, while GPTQ reported about 3.25x compared to FP16 on A100 and 4.5x on A6000 by fitting a 175-billion parameter model on a single GPU (both based on abstracts).

The 0.31.0 documentation cites fp8_per_tensor, fp8_per_block, mxfp8, mxfp4 as the method for online quantization. W8A8 for MXFP8 runs only on Blackwell (SM 100) or newer, and other GPUs fall back to a path where only weights are 8-bit.

3. Measuring it yourself: Online FP8

Method

• The same bf16 checkpoint was loaded twice: once as is, and once with --quantization fp8.
• vLLM 0.31.0 issues a warning for this option: 「--quantization fp8 is no longer recommended for online quantization, so use fp8_per_tensor instead.」 Only the name changed, and the behavior is the same (one scale per tensor). Quality comparisons were conducted on the server relaunched with the new name.
• The workload is the same as in Episode 2 (256 input tokens, 128 output tokens, warming up the first of four batches for each concurrency level).
• Output comparison: Twelve fixed prompts (six copying text verbatim, six stringing together random words) were used to generate 128 tokens via greedy decoding, and output hashes were compared per request.
• Proxy quality metric: Token-level log probabilities of four self-written English texts (338 tokens in total) were obtained (echo + logprobs) to calculate perplexity.

Results

• Why weights became 63% instead of half: In this model, an embedding of vocabulary 151,669 × 1,024 (about 0.3GB) is tied with the output layer, and this layer along with the normalization layers remains in bf16. The smaller the model, the larger this share becomes, reducing the effectiveness of quantization.
• Throughput gains decreased as batch size grew (1.36 → 1.24 → 1.08x). This is exactly as explained in Section 1. Inter-token latency for single-user usage shortened by 27% from 5.6ms to 4.1ms.
• Outputs differ. The six copying prompts matched bf16 down to the letter, while all six random-word prompts diverged starting from the first few tokens. Where multiple candidates have similar probabilities, minor numerical differences cause different tokens to be chosen. The impact of this difference on task accuracy was not measured this time.
• Perplexity increased by 1.2% (36.03 → 36.46). Out of the four texts, two actually decreased slightly (228.5 → 223.5 for the 54-token text, 19.07 → 18.80 for the 55-token text), while it increased from 19.63 to 20.24 in the longest 190-token text. Since 338 tokens is a small sample, a difference of around 1% should be interpreted merely as "it didn't get significantly worse."

3-1. 2026-10-27 Remeasurement: Chat models and pre-quantized checkpoints

Using the official vLLM 0.32.0 container, we compared online FP8 and the 8-bit GPTQ version based on the chat-optimized Qwen3-0.6B, and loaded the 4B model in AWQ 4-bit onto the same GPU. The workload and perplexity methodology are the same as before.

Throughput is in tok/s, and parentheses show ratios compared to bf16.

• GPTQ 8-bit and online FP8 were similar. For single-user usage, GPTQ was slightly faster (244.3 vs. 237.9), but as concurrency increased, FP8 took the lead (3,148 vs. 2,997 at concurrency 64). This is likely because GPTQ keeps only weights at 8-bit while calculation is 16-bit (W8A16), whereas FP8 on this GPU computes even activations in 8-bit. In both cases, vLLM logged that the Marlin kernel was selected (GPTQ).
• Pre-calibrated GPTQ did not pull ahead in perplexity. Across the four texts in this article, FP8 was actually 1.4% lower while GPTQ was 1.6% higher. Given the small sample size of 338 tokens, this should be interpreted simply as "both are largely indistinguishable from bf16." The extraction and tool-calling tasks in Episodes 4 and 5 scored identically at 18/18 and 36/36 across all three approaches.
• AWQ 4-bit fits a 4B model into 8GB. In bf16, weights alone take up about 8GB, making it a model that cannot fit onto this GPU. Instead, the KV capacity shrank to 15,424 tokens, causing the median time to the first token at concurrency 64 to increase up to 3.1 seconds (due to the queue). Perplexity was lower than the 0.6B model (21.02), and in the extraction task of Episode 4, it produced unprompted unfenced JSON all 18 times without enforcement. Whether to run a small model with high concurrency or a large model in 4-bit is determined by concurrency and quality requirements.

4. 8-bit KV cache

--kv-cache-dtype fp8 stores the KV cache in 8-bit instead of weights. In previous measurements using a 1.7B model on the same GPU, the KV capacity expanded from 22,048 tokens to 41,920 tokens (bytes per token exactly halved), pushing back the point where long inputs ran out of KV memory and caused queuing (Episode 14). This time, only the same perplexity test and greedy output comparison were added.

• The KV space being 1.74x rather than exactly double seems to be due to slight differences in remaining memory for KV across servers (unverified as the "memory for KV" line for this server was not recorded). In previous actual measurements with 1.7B, the bytes per token were exactly halved.
• Perplexity increased by 1.3%, similar to weight FP8.
• For greedy outputs, 5 out of 6 copying prompts matched bf16, while one started writing different text: "The scheduler is a class with a method schedule() that …". Since all 6 copying prompts were identical with weight FP8, the KV side changed the outputs more in this test. All 6 prompts chaining random words were different.

5. Takeaways for Operations

• Conversational services with lower concurrency benefit more noticeably from weight quantization. The 1.36x gain at concurrency 1 dropped to 1.08x at concurrency 64. For batch jobs maximizing throughput with large batches, batching and KV capacity take precedence over quantization.
• Quantization changes the model. Since outputs change, you must run your task's evaluation set before and after enabling it. Perplexity is merely a quick primary check.
• When downloading pre-quantized checkpoints, check the source and format. Python pickle (.bin, .pt) weights can execute code upon reading, so choose the safetensors version and record the hash.
• Names change as versions update. In 0.31.0, --quantization fp8 is being deprecated in favor of fp8_per_tensor along with a warning, and 0.32.0 introduced a feature to requantize a checkpoint's quantization layers to a different precision (unmeasured).

6. Unverified Items

• GGUF and bitsandbytes were unmeasured because they require downloading new checkpoints or packages. Comparisons of GPTQ 4-bit and AWQ with bf16 of the same size (4B bf16 does not fit on this GPU) were also unmeasured.
• The impact of FP8 on task accuracy (Q&A, code generation, etc.) was unmeasured. General evaluation was meaningless for this model as it was used for reranking.
• Gains on larger models (expected to be higher due to a larger weight share) were unmeasured.

References

• vLLM Documentation, Quantization, Online Quantization, Quantized KV Cache (0.31.0·0.32.0): https://docs.vllm.ai/en/latest/features/quantization/
• Micikevicius et al., "FP8 Formats for Deep Learning", 2022. https://arxiv.org/abs/2209.05433
• Lin et al., "AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration", MLSys 2024. https://arxiv.org/abs/2306.00978
• Frantar et al., "GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers", ICLR 2023. https://arxiv.org/abs/2210.17323
• Previous Actual Measurement: Re-measuring vLLM directly with a smaller model