A Float Is a Promise to Be Close — Sums, Comparisons, IDs, Variance, Percentages

In one line

A float is not wrong; it writes numbers close, in a fixed way. The problem is the place where that "close" collides with a service's rules. The sum changes with the order, == becomes false, a 64-bit ID in a browser turns into its neighboring ID, a variance goes negative, and percentages add up to 99. This module checks those places one by one with numbers.

Why this was needed

A report comes in that the total of a settlement batch differs by a few won on every rerun. The order in which the pieces were added in parallel was different each time. On the admin screen, clicking an order opens a different order. The server sent the ID as a JSON number, and the browser read it as a double. The standard deviation on the response time dashboard is sometimes NaN. While computing the mean of squares of epoch milliseconds, the code took the square root of a negative variance. None of these raises an exception.

How it works

0.1 is not 0.1. The Python tutorial says that on almost all platforms a float is an IEEE 754 binary64 with 53 bits of precision, and shows that the actual value stored for 0.1 is 0.1000000000000000055511151231257827021181583404541015625, using Decimal.from_float(0.1). As the decimal documentation says, Decimal(float) converts a binary value to decimal without loss, so this is the most honest window onto a float's actual value. float.hex() shows the same value as a mantissa and an exponent.

A sum depends on the order. Every addition rounds the result to 53 bits, so next to a large value the low digits of a small value get cut off. That is why the same sequence gives different values when added from the front, from the back, or sorted. math.fsum carries several partial sums and tracks the lost digits. One thing to be careful about is the built-in sum(). According to the functions documentation, since 3.12 float summation uses a more accurate algorithm that is closer to commutative. That is why, in this lab, "the sum added from the front" is defined as a for loop using +=. It also means that code that ran in another language or on 3.11 and earlier can give different numbers when moved over.

For comparison, use math.isclose instead of ==. The defaults are rel_tol=1e-09 and abs_tol=0.0, and the documentation says that when comparing with 0, isclose(x, 0) is always false for nonzero x, so you should give an appropriate abs_tol. If you forget abs_tol in a check that a balance must be 0, it fails because of a balance of 1e-16.

Integers beyond 2^53. MDN's Number.MAX_SAFE_INTEGER is 9007199254740991 (2^53 − 1), and it notes that MAX_SAFE_INTEGER + 1 === MAX_SAFE_INTEGER + 2 is true. Section 6 of RFC 8259 also says that among implementations that use binary64, values match exactly for integers in the range [−(2^53)+1, 2^53−1]. Outside that range there is no such agreement. If you send a 64-bit ID that grows around 2^60, like a Snowflake ID, as a JSON number, the moment the receiver reads it as a double, nearby IDs become the same value. The fix is to send the ID as a string. On the Python side, json.loads reads integers as int, so it is fine, and a server-side test will not reveal it.

Rounding. round() sends a tie to the even side when two candidates are equally close, and the documentation explains that round(2.675, 2) being 2.67 and not 2.68 is not a bug but because 2.675 cannot be written exactly as a float. To pin the rounding mode down as a policy, create a Decimal directly from a string and use quantize(..., rounding=ROUND_HALF_UP). That the sum of lines rounded one by one differs from the total rounded once is not an error but a difference in the contract.

Variance. The textbook formula E[x²] − E[x]² is the difference of two large numbers. The Algorithms for calculating variance entry says this cancellation is especially bad when the standard deviation is small compared with the mean, and gives an example where shifting 4, 7, 13, 16 by 10^9 makes this formula give −170.67 (the true value is 30). Welford's (1962) method, introduced in the same entry, avoids this cancellation by updating the mean and the sum of squared deviations in one pass.

Percentages. If you round each share separately, the sum becomes 99 or 101. The largest remainder method (Hamilton's method) first hands out the integer part of each share, and then gives out the remaining slots one at a time in descending order of fractional part.

What it looks like in the field

• If the order ID in the server log and the ID shown on the screen differ only in the last two or three digits, suspect a double first. Near 2^60 the gap between neighboring doubles is 256, so the last digits get smeared within that width.
• An inquiry that the line sum and the total of an invoice differ by a few cents is often not a calculation error but a problem of a contract that never decided "where to round." The smallest integer unit of money, rounding policy, and allocation are covered in depth by the fde-bank-money lab of the "The Language of Banking" course, and which item gets the leftover 1 won is covered by dk-com-allocate in the "A Hundred Orders, Ninety-Eight Settlements" course.
• A standard deviation on a monitoring screen that is sometimes blank can be the trace of a square root of a negative variance becoming NaN. It goes away if you subtract a reference point from the values or switch to Welford.

What you will do in the next lab

You will look at the actual values of a few decimals, add a 3,010-line money flow in four orders, and count the pairs where == and isclose answer differently. You will count how many pairs of 64-bit order IDs collide as doubles and build a function that sends them as strings, pin down a rounding policy with Decimal, compute variance with Welford, make percentages that sum to 100 with largest remainder, and then gather it all into a single dashboard response.