From Phone Video to Scores: The Four Tiers and Time Budget of TT Score

Status: Draft (2026-10-13), reflecting measured data. The accuracy and timing metrics in this post are measured values, with measurement dates and conditions noted in each table. Academic figures are explicitly marked as "Paper," and unmeasured items are marked as "Unmeasured." As research is ongoing, these numbers are subject to change. Rule of thumb: Measured data is the default; unmeasured items are marked as "Unmeasured."

Table of Contents

• 1. What TT Score Aims to Do
• 2. Map of the Four Tiers
  • Why Split into Tiers?
  • What Was Chosen and Excluded in Each Tier
• 3. Time Budget: 16.7ms and 1.5 Seconds
  • Actual Time Spent per Frame
  • Time to Decision
• 4. Data Flow Between Tiers
• 5. Current Standing: The October 13, 2026 Baseline
• 6. Future Direction
• 7. Series Map
• References

1. What TT Score Aims to Do

If you place a smartphone on a tripod beside a ping-pong table, the scoreboard on the screen advances automatically with every rally. When a call is ambiguous, it displays "Review Needed," allowing a human to tap and correct it once. These corrections become the training data for the next iteration.

While simple in concept, updating a single point requires the machine to determine all of the following:

1. Where is the ball on the screen? Tracking a 4cm-wide ball traveling at speeds exceeding 100 km/h in every single frame.
2. Where is the table on the screen? To know if the ball has hit the table and which half it landed on, the four corners of the table are required.
3. What happened? Extracting events from the ball's trajectory, such as "bounce," "net," "out," and "rally end."
4. So, who won? Applying the sequence of events to table tennis rules to determine the winner and the reason.

This series explores the models and rules that answer these four questions one by one. This post serves as the overall roadmap.

2. Map of the Four Tiers

Why Split into Tiers?

The primary reason is that the adjudication rules must be correct before the modelsNo matter how accurate the ball coordinates are, misinterpreting rules like "the serve must bounce once on your side and once on the opponent's side" leads to incorrect scores. Therefore, Tiers 2 and 3 are decoupled from the models as pure Python functionsAll rules can be tested using synthetic trajectories and events without any video, and swapping out the ball model leaves the referee untouched.

The second reason is to isolate failure pointsIf a single monolithic model processes video end-to-end to output scores, it is impossible to tell whether an error stems from failing to detect the ball or misapplying the rules. By dividing into tiers, the same evaluation can be run with three different inputs: "model-detected ball," "ground truth ball," and "ground truth bounce," allowing us to measure which tier generates the error. Part 5 demonstrates using this method that "most remaining errors stem from event extraction, not ball detection."

The third reason is licensing and costEach tier can use the smallest possible tool. Ball detection only requires a 1.49-million parameter model, and the referee requires no GPU.

What Was Chosen and Excluded in Each Tier

3. Time Budget: 16.7ms and 1.5 Seconds

Real-time scoreboards face two time constraints.

• Frame Budget. A 60fps video delivers a new frame every 16.7ms. Failing to process within this window causes frame drops. For 30fps, it is 33.3ms.
• Decision Budget. The time allowed from the end of a rally until the score updates. The target is set within 1.5 seconds.

Actual Time Spent per Frame

The threefold difference in ball detection time between the two rows is core to the real-time design. BlurBall takes 3 frames and outputs 3 heatmaps. For recordings, windows can be shifted by 3 non-overlapping frames, but real-time processing requires running a fresh window on "the latest 3 frames including the newly arrived frame" every time a new frame arrives, using only the final heatmap. This choice triples the GPU workload in exchange for keeping latency within a single frame interval. Using the middle heatmap would look at both preceding and succeeding frames but would require waiting one extra frame; the empirical performance difference between the two approaches was under 1 percentage point (approx. 300 sample frames).

In summary, on an empty card, a single session consumes about 63% (10.5 / 16.7ms) of the 60fps budget. Sharing the same card with other tasks pushes a single window past 40ms, failing to maintain 30fps let alone 60fps. Therefore, the system drops lagging windows without queuing them. Only one slot is kept for the next window to process, and if the GPU falls behind, new windows overwrite that slot. Queuing would cause latency to grow indefinitely. In the 2026-10-11 measurements, the proportion of frames dropped due to processing speeds falling below 30fps (22–28% of submitted frames) did not result in increased latency.

The current real-time demo has the browser send JPEGs downsized to 512×288 (quality 0.7, measured 12–18KB) at 30fps intervals. Sending raw video would increase bandwidth requirements tenfold and merely increase the server's downscaling burden. In the development environment, the round-trip time from browser submission to response was a median of 65ms, with the 90th percentile at 84ms. Operational performance over the public internet remains unmeasured due to varying network hops.

Time to Decision

The event tier uses the rule: "A rally ends if no new bounce occurs for 1.5 seconds after the last bounce." 1.5 seconds covers 98% (1,066 out of 1,087) of bounce intervals within rallies across 12 real matches. However, this is the worst-case path. An "out" call triggers when the ball lands off the table, and an "out of play" end triggers immediately if the ball disappears from the screen.

Decision latency measured on October 13, 2026, using the actual deployment path (Section 4 below) is as follows. Decision latency is calculated as the timestamp of the event generating the decision minus the timestamp of the human-labeled rally end.

This represents only the algorithmic waiting time, excluding computation and network transit times.

4. Data Flow Between Tiers

Splitting into tiers makes the "contracts" between tiers critical. The actual values exchanged are as follows:

Two aspects stand out. First, Tier 2 receives ball coordinates only Video pixels do not cross past Tier 1. Second, bounce events carry "horizontal velocity." The referee uses this value to distinguish between "balls heading toward the net like a serve" and "balls returning to the opponent." This single numerical value significantly reduced false scoring in real matches (Part 5).

5. Current Standing: The October 13, 2026 Baseline

Evaluated using 12 actual matches from the public dataset OpenTTGames (5 for training and 7 for testing) along with the Extended OpenTTGames labels, which add rally end causes on top of them. The targets are 275 rallies with known servers, and the primary metric is rally winner accuracy.

Conditions: Ball coordinates use BlurBall pre-trained weights as-is (no fine-tuning), table corners are obtained from the dataset's segmentation masks, engine version 2026-10-13. Only the engine was re-run on a laptop CPU.

There are four caveats to keep in mind when reading these numbers.

1. Even the same engine behaves differently depending on the pipeline. Recorded scoring and the live pipeline differed by 3 rallies on clean footage, but their rankings flipped under dark or blurry conditions. Therefore, all subsequent experiments report both pipelines.
2. Scores are within-sample. The rule engine was tuned while observing these 12 matches. The same applies to the 7 test matches, making this accuracy more optimistic than performance on unseen footage.
3. One gym, one angle. OpenTTGames was filmed in a single venue with a fixed side camera at 120fps (downsampled to 60fps). Performance with mobile phones, local table tennis club lighting, and different angles has not been measured. Since the list of 26 BlurBall training videos is not public, it is unknown whether they overlap with OpenTTGames.
4. Changing the corners changes the numbers. Recorded scoring using corners obtained with SAM 2.1 on 2026-10-12 was 93.1% (256/275). Since that corner file was lost, the baseline was re-established using the dataset mask corners. How much corner quality affects the score is covered in Parts 3 and 5.

6. Future Directions

Most of the remaining errors stem from event extractionrather than ball detection (Part 5). Therefore, the next phase of focus is less on "models that find the ball better" and more on "signals that distinguish bounces and hits and filter out rule-violating sequences." The model investigation from 2026-10-13 has a three-line authority structure. Below is the investigation and design; effectiveness has not been measured yet (in progress).

1. Perception. Output probabilities and "health" (0–1, indicating whether this signal is currently reliable) separately for each signal. Ball detection uses BlurBall constantly, turning on a secondary ball model only during occlusions or darkness. Add racket/pose, audio (onset + surface classification, with synchronization and health assessment taking priority), and table plane.
2. Events → Umpire. Replace the current rule-based decision logic with Constrained Viterbi decodingincorporating table tennis rally sequences as allowed transitions. If the score difference between the best and second-best paths is small, trigger "Review Required" (Part 5, Section 5).
3. Slow Review. Send clips of rallies requiring review to a vision-language model for cross-verification, and flag them only if it disagrees with the umpire (Part 8).

The five candidates selected by the same investigation are Constrained Viterbi with rule grammar, precise event localization models (such as AdaSpot), audio surface classification and audio-video synchronization, RacketVision (5-point racket tracking), and occlusion-resilient ball tracking (TOTNet). We will evaluate them one by one using the same baseline (Section 5) and the same two pipelines, and add them to this series.

7. Series Map

A Word from TT. Finding the ball well and scoring accurately are two different problems. This series traces the gap between them with numbers.

References

• Code paths in this series are based on the TT Score engine folder. ttscore/ is the judging engine (pure Python), live/ is the live worker, batch/ is the recorded video worker, and experiments/ is the experimental code.
• Measurement raw data and methods: Repository docs/tt/PERFORMANCE.md 2.1·2.9.8, Live Demo Design Document Section 9, Experiment Ledger E00 (2026-10-13).
• OpenTTGames: https://lab.osai.ai/ (CC BY-NC-SA 4.0)
• Extended OpenTTGames: https://arxiv.org/abs/2512.19327, https://github.com/moamal01/table_tennis_data (CC BY-NC-SA 4.0)
• BlurBall: https://arxiv.org/abs/2509.18387, https://github.com/cogsys-tuebingen/BlurBall (Code MIT)