Doing int8 Arithmetic by Hand

Goal

Do the arithmetic of integer quantization yourself using only the standard library. Starting by pinning down the rounding rule, you build symmetric and asymmetric quantization, measure the error of the folded-and-unfolded values, count how badly one outlier ruins the scale, compare per-tensor and per-row units side by side, multiply codes with integers only, and finally measure what that error becomes in the probabilities after passing through the softmax.

Why it matters

A quantization tool is one line. If you do not know what arithmetic happened inside, when accuracy drops all you can do is change options and run it again. Once you have seen on a list of eight values what the scale is decided by, which way the rounding goes, and what changes when you narrow the unit, you can point to the same places in large models. This lab does not use a tool. The system Python of this Pod has no numpy (numpy exists only inside /opt/onnx-lab/bin/python), and it does not call a model either. So statements like "model X loses N percent of accuracy at int8" are not made here. You use only numbers measured from the lists and matrices you built. What is difficult is not the formulas but the details. If it is not fixed which way 0.5 goes, whether to clip values outside the range, and what to take the maximum of for the scale, the codes are off by one slot even from the same input. The grader does not trust the explanations you wrote down. It actually imports your module, pokes at the functions with different inputs every time, and checks them against values it computes separately. Most of the checks are between integer arrays, so there is no wobble.

Steps

1. In /root/work/tf-quant/quant.py, create QMAX = 127, UMAX = 255 and round_half_even(x), round_half_away(x) and rounding_gap(values). You confirm with your own eyes where the two rounding rules part ways.
2. Add sym_scale(values), quantize_sym(values, scale) and dequantize_sym(codes, scale) to build symmetric quantization. The scale is max(|x|) / 127.
3. Add affine_params(values), quantize_affine(values, scale, zero_point) and dequantize_affine(codes, scale, zero_point) to build asymmetric quantization. The scale is (max - min) / 255.
4. Add levels_used(codes) and error_stats(original, restored) to build a yardstick that measures the error.
5. Add outlier_effect(values, outlier) so that it measures what one large value does to the other values.
6. Add quantize_tensor(matrix), quantize_rows(matrix) and granularity_gap(matrix) to compare per-tensor and per-row units.
7. Add transpose(matrix), int_matmul(left_codes, right_codes), float_matmul(left, right) and quant_matmul(left, right) to multiply matrices with integers only.
8. Create WEIGHTS, OUTLIER, QUERIES, KEYS and softmax(scores) and attention_shift(queries, keys), and record the results in /root/work/tf-quant/quant_report.json and /root/work/tf-quant/quant_report.md.

Notes

• Execution contract: the grader imports /root/work/tf-quant/quant.py as a Python module and uses QMAX, UMAX, round_half_even, round_half_away, rounding_gap, sym_scale, quantize_sym, dequantize_sym, affine_params, quantize_affine, dequantize_affine, levels_used, error_stats, outlier_effect, quantize_tensor, quantize_rows, granularity_gap, transpose, int_matmul, float_matmul, quant_matmul, softmax, attention_shift, WEIGHTS, OUTLIER, QUERIES and KEYS directly. It does not run it as a script, so if __name__ == "__main__" is not needed.
• round_half_even(x) is exactly Python's default round. The position at exactly 0.5 goes to the even side — round(0.5) is 0, round(1.5) is 2 and round(2.5) is 2. Run it yourself to confirm. The return value is an integer.
• round_half_away(x) sends the position at exactly 0.5 to the side away from 0. If you write it as the single line math.floor(x + 0.5), it is wrong for negatives — -1.5 must go to -2.
• rounding_gap(values) collects and returns only the values on which the two rules part ways, in the order received. Ending in 0.5 does not always make them part ways.
• sym_scale(values) is max(|x|) / QMAX. If all the values in the list are 0, it cannot be divided, so return 1.0.
• quantize_sym(values, scale) divides by the scale, rounds with round_half_even, and then clips to the range from -QMAX to QMAX. dequantize_sym(codes, scale) only multiplies by the scale.
• affine_params(values) returns (scale, zero_point). scale = (max - min) / UMAX, and zero_point = round_half_even(-min / scale) clipped to the range 0 to UMAX. If the maximum and minimum are equal, it is (1.0, 0).
• quantize_affine clips round_half_even(x / scale) + zero_point to the range 0 to UMAX. The clipping is actually used — if both ends are pushed by one slot in the rounding, the sum becomes 256. dequantize_affine is (code - zero_point) * scale.
• levels_used(codes) is the number of distinct codes. error_stats(original, restored) is a dictionary with the three keys max_abs, mean_abs and max_rel. max_rel is the maximum absolute error divided by the maximum absolute value of the original — because dividing by each value individually makes it jump to infinity near 0. If the maximum absolute value of the original is 0, max_rel is 0.0.
• outlier_effect(values, outlier) measures the case of folding only values and the case of folding values + [outlier], and compares only the original values positions. The keys returned are the six clean_scale, dirty_scale, clean_levels, dirty_levels, clean_max_abs and dirty_max_abs.
• quantize_tensor(matrix) returns (배율, 코드 행렬) and quantize_rows(matrix) returns (배율 목록, 코드 행렬) (the placeholders are the scale or the list of scales, and the code matrix). The keys of granularity_gap(matrix) are the six tensor_max_abs, row_max_abs, tensor_worst_row_rel, row_worst_row_rel, tensor_worst_row_levels and row_worst_row_levels. worst_row_rel is the largest value after getting max_rel from the error_stats of each row, and worst_row_levels is the smallest value after getting levels_used for each row.
• int_matmul(left_codes, right_codes) multiplies with integers only. No real numbers may get mixed in along the way. quant_matmul(left, right) folds the left side per row and the right side per column, multiplies with int_matmul, and unfolds with acc * left_scale * right_scale. The return value is a four-element tuple (복원 행렬, 정수 누적 행렬, 왼쪽 배율 목록, 오른쪽 배율 목록) (the placeholders are the restored matrix, the integer accumulation matrix, the list of left scales and the list of right scales).
• softmax(scores) takes one list and returns a list of probabilities. attention_shift(queries, keys) gets the scores with float_matmul(queries, transpose(keys)) and the approximate scores as the first value of quant_matmul(queries, transpose(keys)), and then returns the four keys score_max_abs, prob_max_abs, argmax_changed and top_prob_max_abs. This lab uses the scores as they are.
• The shape of the step 8 materials: WEIGHTS is 6 rows by 8 columns, QUERIES is 4 rows by 8 columns and KEYS is 5 rows by 8 columns, and every value has an absolute value of at most 2.0. WEIGHTS must differ greatly in width from row to row — the maximum absolute value of the largest row must be at least 10 times that of the smallest row for per-tensor and per-row units to part ways. OUTLIER is one value with an absolute value of at least 20.0. Choose the numbers freely.
• The outlier key matrix of the step 8 report is a copy of KEYS with only position [0][0] replaced by OUTLIER. Leave the original KEYS as it is.
• This Pod has no internet. pip install does not work and import numpy does not work in the system Python either (numpy exists only inside /opt/onnx-lab/bin/python). math alone is enough.
• Official documents: Inference with integer arithmetic only · LLM.int8() · Attention Is All You Need · Python math
• Common mistakes: writing round_half_away as the single line math.floor(x + 0.5), not clipping values outside the range, taking the scale from the maximum (not the absolute value), dividing the relative error of error_stats separately for each value, counting the outlier's own error in outlier_effect, folding the right side per row instead of per column in quant_matmul, and multiplying by the scale ahead of time inside int_matmul.

Pin down the rounding first

In /root/work/tf-quant/quant.py, create QMAX = 127, UMAX = 255 and round_half_even(x), round_half_away(x) and rounding_gap(values). The first is exactly Python's default round, sending exactly 0.5 to the even side, and the second sends it to the side away from 0. rounding_gap collects and returns only the values on which the two rules part ways, in the order received.

Run python3 -c "print(round(0.5), round(1.5), round(2.5))" first. You get 0 2 2. If you write round_half_away as the single line math.floor(x + 0.5), it is wrong for negatives — -1.5 must go to -2, but that formula gives -1. Get the lower value with math.floor(x), look at the fractional part separately, and split only the positions at 0.5 by sign. Both functions return integers.

Fold and unfold with symmetric quantization

Add sym_scale(values), quantize_sym(values, scale) and dequantize_sym(codes, scale). The scale is max(|x|) / QMAX, and it is 1.0 if the values are all 0. When folding, divide by the scale, round with round_half_even, and clip to the range from -QMAX to QMAX.

You must take the scale from the maximum absolute value, not the maximum, so that the negative side also fits in the range. The clipping is one line: max(-QMAX, min(QMAX, code)). The unfolding function only multiplies by the scale, so it is one line — it is normal that the unfolded value does not equal the original. What you lost falls within half of the scale.

Use all 256 slots with asymmetric quantization

Add affine_params(values), quantize_affine(values, scale, zero_point) and dequantize_affine(codes, scale, zero_point). The scale is (max - min) / UMAX, and the zero point is round_half_even(-min / scale) clipped to the range 0 to UMAX. If the maximum and minimum are equal, it is (1.0, 0). When folding, clip round_half_even(x / scale) + zero_point to the range 0 to UMAX.

If you forget the clipping, it actually blows up here — if both ends are pushed by one slot in the rounding, round(x/scale) + zero_point becomes 256 and goes beyond the uint8 range. The unfolding formula is (code - zero_point) * scale. Thanks to this formula, the real number 0 comes back without error — that is the reason the zero point exists.

A yardstick for how far it has moved

Add levels_used(codes) and error_stats(original, restored). The first is the number of distinct codes, and the second is a dictionary with the three keys max_abs, mean_abs and max_rel. max_rel is the maximum absolute error divided by the maximum absolute value of the original, and it is 0.0 if that maximum absolute value is 0.

If you divide the relative error by each value individually, it jumps to infinity at values near 0. So you divide by the width the list holds — the scale is decided by that width, so it is also what to compare against. levels_used is the single line len(set(codes)). How many slots are being used even though there are 256 is the subject of the next step.

What one large value does to the rest

Add outlier_effect(values, outlier). Measure the case of folding only values and the case of folding values + [outlier], and compare only the original values positions. The keys returned are the six clean_scale, dirty_scale, clean_levels, dirty_levels, clean_max_abs and dirty_max_abs.

Fold the whole list with the outlier attached and then cut out only the first len(values) items to compare. If you count the outlier's own error, the story is reversed — what gets ruined is not the outlier but the ordinary values next to it. See with your own eyes what dirty_levels drops to. How many slots are you using even though there are 256?

One scale for the tensor versus a scale per row

Add quantize_tensor(matrix), quantize_rows(matrix) and granularity_gap(matrix). The first two return (배율, 코드 행렬) and (배율 목록, 코드 행렬) respectively (the placeholders are the scale or the list of scales, and the code matrix). The keys of granularity_gap are the six tensor_max_abs, row_max_abs, tensor_worst_row_rel, row_worst_row_rel, tensor_worst_row_levels and row_worst_row_levels.

worst_row_rel is the largest value after getting max_rel from the error_stats of each row, and worst_row_levels is the smallest value after getting levels_used for each row. If you look only at the maximum absolute error, the difference between the two methods is barely visible — because that value is decided by the largest row. How a small row gets squashed shows up in the relative error and the number of slots used.

Multiply matrices with integers only

Add transpose(matrix), int_matmul(left_codes, right_codes), float_matmul(left, right) and quant_matmul(left, right). int_matmul must have every product and every sum be an integer. quant_matmul folds the left side per row and the right side per column, multiplies with integers, unfolds with acc * left_scale * right_scale, and returns (복원 행렬, 정수 누적 행렬, 왼쪽 배율 목록, 오른쪽 배율 목록) (the placeholders are the restored matrix, the integer accumulation matrix, the list of left scales and the list of right scales).

To fold the right side per column, pull out the columns with transpose, apply quantize_sym to each column, and turn it back with transpose. Never multiply by the scale inside the accumulation — multiply only once at the end. That is why the error of a matrix multiplication is not something that swelled up in the accumulation but something that already arose when folding at the start. The grader checks the integer accumulation matrix as it is, so even one slot off is caught.

What remains after the softmax

Create WEIGHTS (6 rows by 8 columns), OUTLIER (an absolute value of at least 20.0), QUERIES (4 rows by 8 columns) and KEYS (5 rows by 8 columns), and softmax(scores) and attention_shift(queries, keys). In WEIGHTS, the maximum absolute value of the largest row must be at least 10 times that of the smallest row, and every value in the three matrices has an absolute value of at most 2.0. Then write sym_scale, sym_max_abs, sym_mean_abs, sym_max_rel, sym_levels, affine_scale, affine_zero_point, affine_max_abs, affine_levels, outlier_clean_scale, outlier_dirty_scale, outlier_clean_levels, outlier_dirty_levels, outlier_clean_max_abs, outlier_dirty_max_abs, tensor_max_abs, row_max_abs, tensor_worst_row_rel, row_worst_row_rel, tensor_worst_row_levels, row_worst_row_levels, matmul_max_abs, matmul_max_rel, clean_score_max_abs, clean_prob_max_abs, clean_argmax_changed, dirty_score_max_abs, dirty_prob_max_abs and dirty_argmax_changed in /root/work/tf-quant/quant_report.json, and write /root/work/tf-quant/quant_report.md in the four sections ## 무엇을 쟀나 ## 대칭과 비대칭은 어디서 갈렸나 ## 이상치 하나가 한 일 ## 행렬 곱과 소프트맥스를 지나면 (the Korean headings mean "What was measured", "Where symmetric and asymmetric parted ways", "What one outlier did" and "After a matrix multiplication and the softmax").

Do not write the numbers by hand; fill them in with values obtained by actually running your own code. sym_* and affine_* are values for the list obtained by flattening WEIGHTS into one line, and outlier_* is the result of calling outlier_effect with that list and OUTLIER. tensor_* and row_* are the values of granularity_gap(WEIGHTS), and matmul_* are the max_abs and max_rel of error_stats comparing the restored matrix of quant_matmul(QUERIES, transpose(KEYS)) with float_matmul(QUERIES, transpose(KEYS)). clean_* is attention_shift(QUERIES, KEYS), and dirty_* is the result of calling it with a matrix that is a copy of KEYS with only position [0][0] replaced by OUTLIER. Do not modify the original KEYS. See which way the score error and the probability error move and write what you saw — do not write by guessing.