Reducing Numbers: FP16, BF16, INT8 Quantization, Post-Training Quantization, Quantization-Aware Training, and Calibration Data

Status: Draft (2026-10-16), partially measured. GPU measurements were performed in a single-use pod on an 8GB class laptop GPU in the home lab (median of iterations after warm-up). Accuracy was compared using a single public dataset OpenTTGames test match (388 frames at 60fps with markers). INT8 was run using PyTorch's post-training quantization on the CPU of the same pod (4-core limit). The principles of formats and calibration were verified with a laptop CPU example (code/02_quant_error.py, around 1 second). Part 2 of the series 「Reading Vision Models 3」. Latency budget is in Part 1.

Table of Contents

• 1. Why Reducing Bits Makes It Faster
• 2. Two 16-Bit Formats: FP16 and BF16
• 3. INT8: Fitting Real Numbers to a Grid
  • How Many Grid Points: Per-Tensor vs. Per-Channel
• 4. Post-Training Quantization (PTQ) and Calibration Data
• 5. Quantization-Aware Training (QAT)
• 6. Our Measurements: FP16 for the Ball Model
  • Speed
  • Accuracy: Does the Ball Position Change?
• 7. Our Measurements: INT8 Post-Training Quantization for the Ball Model (CPU)
• 8. Limitations
• 9. Connection to Our Project
• References

1. Why Reducing Bits Makes It Faster

Most neural network inference time is spent on two things: fetching numbers from memory, and multiplying and adding them. Reducing a number from 32 bits to 16 bits halves the bytes to fetch. Moreover, modern GPUs and phone chips have dedicated units that perform many 16-bit or 8-bit multiplications at once. According to NVIDIA's 2018-generation GPU specifications, INT8 compute throughput is 16x that of FP32, and FP16 is 8x (Wu et al. 2020, Table 1 — specification ratios, not actual measurements).

The trade-off is accuracy. Writing numbers with less precision introduces rounding errors, and a narrow range causes large numbers to overflow.

2. Two 16-Bit Formats: FP16 and BF16

Floating-point numbers are divided into sign, exponent, and mantissa. The exponent determines "how large or small the numbers can get," and the mantissa determines "how fine-grained the precision is."

(code/02_quant_error.out.txt PyTorch values for . FP16 is fine-grained but has a narrow range, whereas BF16 has the same range as FP32 but is coarser. Trying them directly, 70,000 becomes inf in FP16 and 70,144 in BF16, while 1e-8 becomes 0 in FP16 and 1.00117e-8 in BF16. The relative error for a 256×256 matrix multiplication was 3.6×10⁻⁴ for FP16 and 2.9×10⁻³ for BF16.

In training, this range issue is significant. Micikevicius et al.'s "Mixed Precision Training" (2018) solved the problem where gradients smaller than 2⁻²⁴ become zero in FP16 by using loss scaling, multiplying the loss by a large number before dividing and maintaining a separate FP32 copy of the weights. With this approach, ResNet-50's ImageNet top-1 matched FP32 at 75.92 versus mixed precision at 76.04 (Table 1). BF16 has the same range as FP32, allowing it to be used without loss scaling, and Kalamkar et al. (2019) trained ResNet-50 with BF16 to achieve 74.7, matching FP32 (comparing similar evaluation methods, Section 4.1.2).

In inference, pre-trained weights are used, so the range issue is less severe. As long as activation values do not exceed 65,504, FP16 is often sufficient.

3. INT8: Fitting Real Numbers to a Grid

8-bit integers have only 256 slots. Jacob et al.'s (2018) affine quantization writes real numbers r as follows.

r = S · (q − Z)
S: 눈금 한 칸의 크기(실수),  Z: 실수 0 에 해당하는 정수(영점),  q: 0–255 의 정수
범위 [최솟값, 상한] 을 정하면  S = (상한 − 최솟값) / 255,  Z = round(−최솟값 / S)

The key point is setting Z so that the real number 0 is represented precisely as a single integer. Padding filled with zeros and zeros after ReLU are then represented without error. Multiplication is performed by accumulating uint8 × uint8 into int32, multiplying by the scale factor just once at the end, and reducing back to 8 bits, allowing the entire inference to be done solely with integers. The same paper quantizes graphs by folding batch normalization into convolutions beforehand (Part 3).

The main point of this paper is that within the same 33ms budget, the INT8 model was about 10% more accurate than the real-number model (Section 4.2.1, mobile small core). This is because the time saved by lowering precision can be used to run a larger model.

How Many Grid Points: Per-Tensor vs. Per-Channel

If a single S is used for the entire weight tensor, a large value in even one channel will widen the scale to fit that channel, causing the rest to collapse. Specifying a separate S for each output channel (per-channel) eliminates this issue. Trying it directly (256×256 matrix multiplication, weights only in INT8):

The same thing happens in real models. In Krishnamoorthi's (2018) whitepaper, converting MobileNet-v1 to 8 bits without training resulted in a top-1 accuracy of 0.001 for per-tensor scaling (completely broken), while per-channel scaling achieved 0.703 (compared to 0.709 for real numbers) (Table 3). Depthwise convolutions (Part 4) exhibit large variations in value magnitudes across channels, making them particularly vulnerable to this issue.

4. Post-Training Quantization (PTQ) and Calibration Data

Post-Training Quantizationpasses several hundred representative inputs through a trained model to measure the range of activation values per layer (calibration) and determines S and Z from those ranges. No training is required. Nagel et al.'s (2021) whitepaper noted that 500–1,000 unlabeled images are usually sufficient, and reported that following standard procedures kept models within 0.7% of FP32 for W8A8 (8-bit weights and activations).

The difficult part is how far to set the range. Using the minimum and maximum values directly causes the grid to become coarse due to a single rare large value. Narrowing the range by clipping the top end slightly (e.g., the 99.99th percentile) makes the grid finer, but the clipped values become inaccurate. We tested this with two types of activation values (calibration and test samples were drawn separately).

• In (A), the min–max range worsened as calibration data increased. This is because more samples encountered larger extreme values, expanding the range from 23 to 215. The 99.99th percentile remained stable regardless of the number of samples.
• In (B), the opposite is true. When rare large values drive the results, clipping them causes major errors. The "outlier features" of language models are an example of this; Dettmers et al. (2022) showed that removing those 0.1% values causes the model to break down.

Therefore, calibration methods must be evaluated per model. In the table from Wu et al. (2020), PTQ on EfficientNet-B0 yielded 22.3% for min–max, 72.1% for entropy, and 70.9% for the 99.9th percentile (FP32 was 76.9%), while Inception v4 achieved 0.1% for min–max and 79.6% for the 99.99th percentile (FP32 was 79.7%) (ImageNet top-1, 1,024 calibration images, Table 5).

5. Quantization-Aware Training (QAT)

If PTQ falls short, quantization is simulated starting from training. Forward passes calculate values using "quantized-and-dequantized" weights and activations (fake quantization), while backward passes let gradients flow directly as if there were no rounding. In effect, the model learns to shift its weights to fit the grid. In Krishnamoorthi's whitepaper, MobileNet-v1, which collapsed with per-tensor scaling (0.001), recovered to 0.70 with QAT (Table 4), and in Wu et al., EfficientNet-B0 reached 76.95% with QAT, surpassing the best PTQ result (72.1%) (fine-tuned for about 10% of the original training schedule, Table 7). The trade-off is the requirement for a training pipeline and data.

6. Our Measurements: FP16 for the Ball Model

Speed

The TT Score ball model (BlurBall, 3-frame window, 512×288) was measured on an experimental GPU (batch size 1, median of 250 runs after 20 warm-up runs).

On the same GPU, the transformer encoder (EfficientTAM-S, Part 4) was 3.1x faster. The ball model is a convolutional network that maintains high resolution all the way through, so the share of dedicated 16-bit hardware acceleration is relatively small, and element-wise operations like batch normalization and ReLU remain. In Part 3, fusing these with a compiler brought it down to 4.3ms.

Accuracy: Does the Ball Position Change?

On a single test match from OpenTTGames featuring 388 frames at 60fps with ball position markers (all containing a ball), FP32 and FP16 were run on the same inputs using production settings (heatmap of the last frame, connected component analysis with a threshold of 0.5). Results within 5px of the ground truth (based on 512×288) were counted as correct. Inputs compressed and decompressed using JPEG quality 70 (same conditions as sent by the browser) were also run with FP32.

FP16 did not change a single judgment across these 388 frames. The difference in heatmap values was around 0.0001 at the center. In contrast, JPEG compression increased missed balls from 4 to 11 and shifted the maximum position by a third of a frame. Rather than halving precision, compressing the input caused a much larger error. Since production already receives JPEGs, we believe enabling FP16 will not cause significant changes compared to now. However, this is based on a single match and condition, and we did not measure degradation conditions (darkness, blur) or production GPUs (not measured).

7. Our Measurement: Post-Training INT8 Quantization of the Ball Model (CPU)

We could not measure INT8 on the GPU this time because it requires a runtime like TensorRT (Part 3). Instead, we converted the same model to INT8 on the CPU using PyTorch's post-training quantization (FX graph mode, x86 integer engine). Calibration frames were selected from the beginning of the match and evaluation frames from the end, ensuring no overlap.

220 evaluation frames (215 with balls), 146 calibration candidates, limited to 4 CPU cores, showing the median time per window run in batches of 4.

• With 64 calibration frames, INT8 produced the same number of judgments as FP32 (208 hits), but missed two more with just 1 frame. This model was gentle enough not to break down even with a single calibration frame.
• Although the difference in heatmap values was about 50 times larger than FP16 (around the 0.001 level), judgments at the 0.5 threshold barely changed. This is because the heatmap values at the ball's location are generally far from the threshold.
• CPU time was about 1/7th of FP32. However, it is hard to attribute this entire multiplier solely to 8-bit. On the same CPU, the FP32 path might have been unusually slow for this model (a convolution maintaining full resolution throughout), and we did not separately verify the cause (not measured). Even excluding this difference, since GPU FP16 takes 10ms for the same model, CPU INT8 at 72ms is not a viable alternative for a real-time server path. Values on mobile CPUs were not measured.

8. Limitations

• Accuracy comparisons are based on a single public match, side camera, and clean conditions. We did not measure whether 16-bit or 8-bit errors increase in frames where the ball is blurry or dark.
• "Identical judgments" is based on finding blob criteria with a 0.5 threshold. Balls near the threshold (confidence around 0.5) can have their judgments flipped by small errors.
• Paper figures are results from their respective data and hardware, and those multipliers cannot be directly applied to our model.

9. Connection to Our Project

• Production operators currently use FP32. Based on the results in Section 6, FP16 reduces a single window by about 35% without altering judgments. Before enabling it, we plan to run the same comparison once more across multiple matches, degradation conditions, and production GPUs.
• For models migrating to mobile (Part 7), 8-bit becomes the default choice. In this case, calibration data must be selected from actual mobile screens (JPEG, various lighting). As seen in Section 4, the characteristics of the calibration data determine the range.

The next post covers compilers and runtimes that change the execution methodwhile keeping precision intact.

A word from TT. We halved the numbers, but the ball position stayed the same. Compressing the images caused more distortion instead. You never know where errors occur until you measure them!

References

• Micikevicius, P. et al. (2018). Mixed Precision Training. ICLR 2018. https://arxiv.org/abs/1710.03740
• Kalamkar, D. et al. (2019). A Study of BFLOAT16 for Deep Learning Training. arXiv. https://arxiv.org/abs/1905.12322
• Jacob, B. et al. (2018). Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference. CVPR 2018. https://arxiv.org/abs/1712.05877
• Krishnamoorthi, R. (2018). Quantizing deep convolutional networks for efficient inference: A whitepaper. arXiv. https://arxiv.org/abs/1806.08342
• Nagel, M. et al. (2021). A White Paper on Neural Network Quantization. arXiv. https://arxiv.org/abs/2106.08295
• Wu, H. et al. (2020). Integer Quantization for Deep Learning Inference: Principles and Empirical Evaluation. arXiv. https://arxiv.org/abs/2004.09602
• Dettmers, T. et al. (2022). LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale. arXiv. https://arxiv.org/abs/2208.07339
• Public Data: OpenTTGames (CC BY-NC-SA 4.0)
• Example Code and Outputs: In this folder's code/02_quant_error.py, code/02_quant_error.out.txt