Events and Referee: Extracting Bounds from Trajectories and Scoring by Rules, From 66.9% to 93.1%

Status: Draft (2026-10-13), reflecting actual measurements. All accuracies are values measured directly from 12 actual OpenTTGames matches (275 rallies with known servers), with dates and conditions attached to each version. Experiments on learned event detection and rally-end determination redesign are ongoing, so their results are not listed. TT4D and PathCRF figures are from the paper. Writing rule: Measured data is the default, and anything unmeasured is marked as "Unmeasured."

Table of Contents

• 1. From Trajectory to Event
  • Intuition: Bounds are Vs, and Shots are Also Vs
  • Every Constant Has a Basis
• 2. Referee: Does Not Watch "Who Hit the Ball"
  • A Single Number Called Horizontal Speed
  • Postponing Fast Double Bounds
  • Confidence and "Needs Review"
• 3. From 66.9% to 93.1%: What Each Version Changed and Paid
  • And the 2026-10-13 Baseline
• 4. In Which Layer Do Errors Occur? Inserting Ground Truth
  • Remaining Errors
  • Under Poor Conditions
• 5. Future: 3D Extremum Rules and Constrained Viterbi
  • TT4D Formula-based 3D Extremum Rules
  • Constrained Viterbi: Treating Rules as "Allowed Transitions"
  • Ongoing Experiments
• References

1. From Trajectory to Event

What the first layer gives you per frame is (시각, x, y, 신뢰도) alone. To score a match, this must be converted into a sequence of events. The engine outputs four events:

Intuition: Bounds are Vs, and Shots are Also Vs

Screen y increases downward. When a ball drops onto the table and bounces back up, y increases and then decreases. This "V" shape is the basic pattern of a bounce. The problem is that balls hit with a racket on the table also form Vs. Therefore, we check one more thing. A ball that bounces on the table continues flying in the same direction, whereas a ball hit by a racket reverses direction. Across 12 actual matches, 97.9% of true bounds maintained their horizontal direction, while 73.8% of fake bounds within 3 frames before and after a shot reversed their horizontal direction.

# ttscore/events.py detect_events() 에서 줄임
going_down = before.x is not None and c.y > before.y      # 같은 y 가 이어지면 그 앞의 다른 y 와 비교
going_up = n.y < c.y
if (going_down and going_up and table.on_surface(c.x, c.y)
        and c.t - last_bounce_t >= MIN_BOUNCE_GAP_S         # 0.12초 — 같은 바운드를 두 번 세지 않게
        and keeps_direction(i)                              # 앞뒤 가로 이동이 둘 다 2px 이상이고 반대면 타구
        and not near_net_touch(i)):                         # 네트에 걸린 공의 V 는 버린다
    emit(Event(c.t, "bounce", table.ball_side(c.x, c.y), conf=min(p.conf, c.conf, n.conf),
               vx=vx_at(i), vx_in=vx_at(i, after=False)))

The single line "if the same y continues, compare with a different y preceding it" also has a backstory. Heatmap coordinates are integer pixels, so identical values persist in slow sections; counting "sameness" as "descending" generated false bounds on ascending balls.

Every Constant Has a Basis

A weakness of rule engines is that constants are easily set by intuition. Therefore, empirical bases for each constant are documented in the code comments.

Paired with the last line is fill_gaps . If the ball is obscured by the body and goes unseen for 6 frames (0.1 seconds at 60fps) or less, it is interpolated with a straight line, and the confidence of the interpolated points is halved relative to both endpoints. The goal is to automatically route bounds occurring in that segment to "Needs Review."

2. Referee: Does Not Watch "Who Hit the Ball"

The core idea of the referee engine (ttscore/referee.py) is summarized in a single sentence in the introduction: Who hit the ball is not directly observed in the video but inferred from the sequence of bounds. If the ball bounced on side S, the player on side S must hit it next and make it land once on the opposite side. Failing to do so (another bounce on the same side, going out without a bounce, or ending without an event) means side S loses the point. The serve is the only exception, requiring one bounce on the server's side followed by one on the receiver's side.

This allows scoring without "racket contact," which is the hardest event to find in video. Instead, if a single bounce is missed or a fake bounce sneaks in, "whose turn it is to hit" gets completely out of sync. Therefore, most of the referee's remaining rules are mechanisms to filter out fake bounds.

A Single Number Called Horizontal Speed

Bounce events carry the horizontal speed immediately following the bounce vx(table lengths/second). Because it is divided by table length rather than screen pixels, it is independent of camera distance and resolution. The referee uses this number to distinguish three things:

1. Start of a serve. Recognized as a serve if it is a bounce on the scoreboard server's side and the ball moves toward the net at 0.7 table lengths/second or faster. The first bounds of all 271 actual serves were 1.07 or higher, whereas balls on the server's side outside of rallies (bouncing on one's own side before serving, passing the ball to the opponent) were 0.61 or lower, excluding 4 that looked like serves.
2. Fake bounds during rallies. True bounds for side S during a rally are shots hit by the opponent and thus head toward side S's end—away from the net. A "bounce" heading toward the net is discarded as a misread ball passing over the net or a shot. This single rule increased the number of correct rally winners from 235 to 241.
3. Returning the ball. When the receiver tosses the ball back to the server, it bounces from the receiver's side toward the net and lands on the server's side. It looks "like a serve" but starts on the side opposite the scoreboard server, so it is not accepted. A variant accepting both sides increased fake judgments across 5 matches from 3 to 16.

Postponing Fast Double Bounds

If two bounces on the same side are closer than 0.25 seconds, it is impossible to tell whether they are true double bounces or a shot immediately following a bounce misread as a bounce. In actual measurements, 16 out of 39 such judgments (41%) were incorrect. The error rate for remaining judgments was 8%. Therefore, this judgment is postponed until the next event. If the next bounce is on the opposite side (a return lands), the second bounce was fake, so it is discarded and the rally continues. All 9 actual rallies whose judgments changed due to this rule went from incorrect to correct. The trade-off is time: that judgment comes out up to 1.5 seconds later.

Confidence and "Needs Review"

The confidence of a rally is the minimumamong the confidences of its constituent events. An event's confidence is the lesser of its detection confidence and preceding gap confidence. If it falls below 0.6, the scoreboard displays "Needs Review." This means the judgment is output as-is, but flagged for human inspection.

Here, two numbers must be distinguished:

• Needs Review Rate: The proportion of rallies that humans must inspect. High values are tedious.
• Incorrect Without Review: Incorrect judgments without a "Needs Review" flag. Because humans trust and pass them, these are actual costly errors.

Conservatively lowering confidence reduces errors without review while increasing needs review, and vice versa. The history below is also a tug-of-war between these two numbers.

3. From 66.9% to 93.1%: What Each Version Changed and Paid

Conditions (all rows identical): 12 actual OpenTTGames matches, Extended OpenTTGames rally end labels, 275 rallies with known servers, BlurBall coordinates, scored from recordings. All dates are 2026-10-11–12.

Listing the trade-offs paid per version yields the following.

• Version 2. Out of 91 errors in the initial version, 65 were due to "reading shots as bounds" and "rallies not ending." Fixing these reduced no-judgment cases from 10.2% to 1.5%. The trade-offs were an increase in Needs Review from 76.4% to 85.1% and the fact that the 1.5-second end rule can prematurely score points in defensive matches with many lobs (1.5s is the 98th percentile, meaning 2% are longer).
• Version 3. Because the ball center at the bounce moment is one radius higher than the contact point, bounds near distant borders fell outside the band. Widening the band increased winner accuracy, but bounce precision dropped slightly from 0.831 to 0.825. A wider band lets in more fakes.
• Version 4. Replaced corners with SAM 2.1. The trade-off is one segmentation model run at the start of the match and two human taps.
• Version 5. This is where the nature of the system changes. A Needs Review rate of 85% means it is not automated scoring but a "review draft." By further improving judgment accuracy through serve recognition rules and fake bounce filtering, and redesigning confidence penalties, we cut Needs Review by more than half, from 85.8% to 41.5%. As a trade-off, Incorrect Without Review increased from 1.5% (4 cases) to 5.1% (14 cases) . Human workload decreased significantly, but unreviewed errors passed to users tripled.
• Version 6. Caught lets where serves touched the net as net judgments, and postponed fast double bounds. Winner accuracy rose to 93.1%, and Incorrect Without Review dropped back down from 5.1% to 3.6%. There are three trade-offs: fast double bound judgments are delayed by up to 1.5 seconds, one actual labeled serve was misread as a let (flagged as Needs Review), and some noise conditions in synthetic tests worsened slightly (83.8 → 83.2%).

And the 2026-10-13 Baseline

Version 6's 93.1% was measured using SAM corners, but those corner files no longer exist. Starting research anew, we reset the baseline with dataset mask corners and the latest engine, this time utilizing actually deployed real-time paths.(Measured by replaying the recent 2.5 seconds per sample).

Using the exact same engine and ball coordinates, changing only the pipeline resulted in 3 different rallies. This is because recorded video evaluation makes judgments after watching the entire rally, whereas real-time evaluation looks at only the last 2.5 seconds and makes decisions on the fly. Recorded video evaluation inflates real-time performance. Therefore, all future experiments will report both pipelines side by side.

4. Where do the errors occur?: Feeding ground truth

Thanks to the separation of layers, we can run the same evaluation with three different inputs: the model's ball detection, the dataset's ground truth ball coordinates, and the dataset's ground truth bounces. If it still fails when given the ground truth ball, the error is not due to ball detection. If it still fails when given the ground truth bounces, the error is not due to event extraction either.

In v2 and v5, performance improved as ground truth was fed into the system. Therefore, we interpreted that "most of the remaining errors stem from event extraction, rather than ball detection." Indeed, out of 25 false bounces in v5, 11 were hits misread as bounces, 5 were side flips near the net, and 4 were missed bounces.

However, in the latest engine, the model ball outperforms both the ground truth ball and ground truth bounces. This reveals why these numbers should not be interpreted as an "upper bound."

• Ground truth balls only exist near events. OpenTTGames ball markers are present only from 4 frames before an event to 12 frames after (Part 7). Since the rest is empty, ground truth balls cannot properly capture the end of a rally (where the ball disappears).
• Ground truth bounces lack horizontal velocity. The server/return rules defined by the engine utilize the horizontal velocity of bounces. Feeding only ground truth bounces disables those rules.

In other words, ground truth inputs are not "perfect inputs" but rather "different inputs." Because the engine is tuned to the properties of the model ball (such as coordinates extending beyond events and horizontal velocity), comparisons with ground truth inputs should be used solely as a tool to isolate layer-specific errors, not as an upper bound.

Remaining errors

These are the categories of the 27 cases that resulted in errors or no decisions in the 2026-10-13 baseline (recorded video, 275 rallies).

Events (bounces) account for 14 out of the 27 cases, and rally ends account for 8. Broken down by cause of point ending: outs were 92.8% (128 out of 138), nets 92.6% (75 out of 81), winners 84.4% (27 out of 32), and double bounces 75.0% (3 out of 4).

Under degraded conditions

We measured performance by introducing degradation across 7 test matches (57 rallies) (recorded video evaluation, correct winner count).

Even though 30fps barely degrades ball detection (Part 2), it caused a significant drop in score, with incorrect decisions without review being the most frequent. This is because the temporal resolution at the moment of the bounce is halved. This engine tends to fail in ways that conceal incorrect answers. Therefore, we recommend 60fps for recording.

5. Looking ahead: 3D extremum rules and constrained Viterbi

TT4D-style 3D extremum rules

Current rules look for a "V-shape" in the 2D image. Because the y-axis in the image mixes ball height and distance from the camera, changes in viewpoint disrupt the rules (end-of-video camera issue in Part 3).

TT4D (arXiv 2605.01234, ACM MM 2026) lifts ball trajectories into 3D space before detecting events. The rules are surprisingly simple:

• Bounce = Local minimum.
• of ball height z; Hit = Extremumalong the table length axis x (where the ball reverses direction).
• At least 0.2 seconds between extrema of the same type, and extremum position at least 0.3m away from the net.

On data with 3D ground truth (synthetic and TT3D), they reported a bounce F1 of 0.92 and hit F1 of 0.83 within a ±2 frame tolerance (paper does not separately report precision and recall). The authors attribute the lower hit score to the ball being frequently occluded near players.

Compared to our 2D rules, this is the 3D counterpart of the same idea. The "V-shape" corresponds to the local minimum of z, and the "horizontal direction reversal equals a hit" corresponds to the extremum of x. Lifting to 3D makes the rules invariant to viewpoint changes, eliminating the distortion caused by projecting airborne balls onto a 2D plane. However, this introduces a new requirement to reconstruct 3D coordinates from a single camera. Whether this reconstruction can be achieved using our side-view mobile footage is the question for our next experiment (in progress, unmeasured). Since some open-source implementations of this family are licensed under GPL, we plan not to borrow their code but to implement the methodology ourselves.

Constrained Viterbi: Turning rules into "allowed transitions"

Currently, the engine receives events one by one and makes decisions on the fly. When a single false bounce is introduced, it flips "whose turn it is to hit," throwing off all subsequent judgments. Humans look back and correct themselves in such situations: "If this bounce were real, the subsequent sequence would violate the rules. Therefore, this bounce must have been false."

Translating this thought process into computation gives us constrained Viterbi decoding.

1. At each timestamp, we score candidate statessuch as "server-side bounce," "receiver-side bounce," "hit," and "no event." These scores are derived from signals such as ball tracking, audio, and player pose.
2. Table tennis rules are formulated as allowed transitions. Examples include: serve → server-side bounce → receiver-side bounce → hit → server-side bounce … is allowed, whereas two consecutive bounces on the same side without a rally continuation is forbidden (a point must have been scored). Forbidden transitions are assigned very low scores (effectively −∞).
3. Using the Viterbi algorithm, we find a single rule-compliant path with the highest total score. Because it is dynamic programming, execution time is proportional to the length of the rally.
4. If the score difference between the best path and the second-best path is small, that rally is marked as "requires review." This provides a more rigorous confidence metric than our current "minimum confidence threshold."

PathCRF (arXiv 2602.12080, KDD 2026), which applied a similar concept to soccer, kept the model unchanged and only modified the decoding to improve event F1 from 65.34% to 73.14% and rule violations from 2.80% to 0% (paper). We expect an even greater effect in table tennis because its rules are much stricter than soccer's (estimated). Since the PathCRF code is licensed under MPL-2.0, we will not import it, but rather re-implement the method tailored to our engine.

Ongoing experiments

The next experiments do not have results yet, so they are omitted here (in progress).

• Using a small 1D convolutional model (approx. 18k parameters) that looks at a short window of ball trajectories (16 frames back, up to 200ms forward) to output bounces, hits, and net events, and filtering or supplementing rule-based bounces. Evaluating the trade-off between accuracy and latency by varying look-ahead lengths.
• Reducing decision latency by tuning the rally-end "ball disappearance" threshold and fixed-latency confirmation.
• Combining the above two approaches with constrained Viterbi.

We have established our experimental principles: hyperparameters are selected exclusively from validation matches not used in training, and the 7 test matches are evaluated only once at the very end. Furthermore, because the rule engine was refined while observing the test matches, we note when comparing against new methods that the rule baseline's test scores are optimistic.

A word from TT. Umpires don't watch the racket. You can get almost everything right just by counting where the ball lands. However, since a single false bounce disrupts the entire sequence, most of the rules are dedicated to filtering out false positives.

References

• TT4D: https://arxiv.org/abs/2605.01234
• PathCRF: https://arxiv.org/abs/2602.12080
• Extended OpenTTGames (rally end cause labels): https://arxiv.org/abs/2512.19327
• Our measurements: Repository docs/tt/PERFORMANCE.md 2.2·2.6·2.7, Experiment ledger E00 (2026-10-13), Code ttscore/events.py·referee.py·track.py