The Day a Single Number Halted Deployment: 729 in 268,729 and Regex Word Boundaries

Status: Resolved (October 15, 2026, 09:43, comment updated). The facts are gathered from batch gate execution histories, gate log failure messages, and commit logs. The regex comparison table values were generated by running Python directly. Fixing the regex and the culprit finder has not yet been done.

Table of Contents

• TL;DR
• 1. What Happened
• 2. Why "268,729" Matches "729": Word Boundaries
• 3. Why It Took 8.5 Hours: The Batch Gate That Couldn't Find the Culprit
• 4. What Needs to Be Fixed
  • Regex: Exclude digits adjacent to the number in numeric brand names
  • Culprit Finder: Check paths in failure messages, split if unknown
  • Loop: Stop and notify when the same failure repeats
• 5. Lessons Learned from This Incident
• References

TL;DR

1. What Happened

The camera scoreboard is the flagship screen of TT Lab. Since the initial screen was fetching recording, editing, and auto-scoring code it didn't even render upfront, I split it into chunks and deferred their loading. I wrote this in the header comment of that change:

//   정적 import 로 268,729 바이트를 받았다(Fast 3G 첫 그리기 4.7초). …

Writing exact numbers is a good habit. The problem was that this repository has a test preventing specific equipment brand names and internal names from mixing into table tennis playground static files . The brand list includes a numeric name famous for table tennis rubbers. Yes, it is 729 .

BRANDS = re.compile(r"\b(butterfly|tenergy|…|sanwei|729)\b", re.I)

The failure message in the gate log was straightforward.

AssertionError: <re.Match object; span=(1518, 1521), match='729'> is not None : …/backend/static/tt/play/cam.js

2. Why "268,729" Matches "729": Word Boundaries

\b matches a position, not a character. It is the position where one side is a word character (\w — letters, numbers, underscores) and the other side is not.

• , is not a word character. Therefore, the space between , and 7 is a boundary.
• 9 The space after is also not a word character, so the space between 9 and the space is also a boundary.
• Result: A number that looks like "268 thousand 7 hundred 29" to human eyes is seen as two separate words, "268" and "729", by the regex engine.

The same applies to decimal points (v1.729) or hyphens (729-2) as well as thousands separator commas. If you wrap a numeric brand in \b , any text containing that number can trigger a match.

3. Why It Took 8.5 Hours: The Batch Gate That Couldn't Find the Culprit

A gate (full test suite) takes about 20 minutes to run. Therefore, instead of running tests separately for each task, multiple tasks are bundled and tested together, and if they pass, they are deployed together. If they fail, the "culprit finder" isolates the task causing the failure and re-runs the rest.

The culprit finder relied on these clues:

1. Tasks that touched files (test_tt_play) with names matching the failed test name (tests/test_tt_play.py, backend/app/tt_play.py)
2. Tasks that touched the file quoted in the failure message ( .py )

This time, both missed. Nobody modified the failed test file, and the file mentioned in the failure message was .py , not cam.js . Even though the full path of cam.js was clearly present in the failure message , the culprit finder ignored it. The result was: "Gate failed, unable to identify culprit."

And a few minutes later, the outer loop re-ran the same batch . Newly incoming tasks also piggybacked onto the same batch.

All 26 runs featured the same test, the same file, and the same match='729' (verified across all 26 remaining gate logs). The task file updating the comment to "received about 262KiB via static import" was modified at 09:42, and the next cycle started at 09:43 and passed (268,729 ÷ 1,024 ≈ 262.4).

4. What Needs to Be Fixed

Regex: Exclude digits adjacent to numeric brands

I compared two regex patterns using Python.

old = re.compile(r"\b(729)\b")
new = re.compile(r"(?<![\d.,])729(?![\d.,])")   # 앞뒤가 숫자·점·쉼표면 상표가 아니다

The new pattern is not perfect either (it catches0x729 ). Since a single regex cannot perfectly distinguish between "729 used as a brand" and "729 inside a number", it is better to err on the side of caution and make it immediately obvious when a match occurs . This change has not been implemented yet.

Culprit Finder: Check paths in failure messages, split if unknown

• Compare every file path (regardless of extension) mentioned in the failure message against the files modified by tasks. This alone would have identified the culprit immediately this time.
• If it still remains unknown, split the batch in half and run only the test that failed (binary search). A single test takes only a few seconds.

Loop: Stop and notify when the same failure repeats

If the same test fails twice for the same reason, it will fail a third time. In that case, instead of re-running, it should stop and present the failure message to a human. The 26 retries consumed 8.5 hours, during which 18 unrelated tasks were held up waiting.

None of these three changes have been made yet. This article serves as that to-do list.

5. Lessons Learned from This Incident

• Guarding tests can fail for the wrong reasons. The trademark check did its job. It just didn't know how to distinguish between the brand "729" and the number "268,729".
• "Unknown" should be an alert, not a result. Failing to identify a culprit means "A human needs to look at this," not "Let's try again."
• Batches win together and lose together. Batch gates save time, but the moment they fail to isolate the culprit, the entire batch is held hostage.

A Word from TT. 729 is a great rubber, but hiding inside numbers causes trouble! The regex \b doesn't treat commas as walls. And if you stumble twice for the exact same reason, check your shoelaces before running a third time.

References

All materials accessed on October 16, 2026.

• Python re documentation (\b, lookahead/lookbehind (?<!…)·(?!…)): https://docs.python.org/3/library/re.html