Digital Twin and AI Table Tennis Referee: Matching Physics with Real Bounces and Testing the Referee in an Exam Hall with Known Answers

Status: Draft (2026-10-18), Measured. The accuracy in the test hall represents relative comparison values within the simulator. All calculations were performed on a single Mac (Apple M1 Max 10-core, Python 3.9.6, numpy 2.0.2) CPU without using GPUs or clusters. No new packages, images, or data were downloaded. Tuning and the test hall were executed deterministically with fixed seeds (the same command produces the same result), and processing times are the median of 5 runs. Processing times were measured when other tasks were running on the Mac and the load average was 47–76. Actual videos used the public dataset OpenTTGames (120 fps, human-annotated ball coordinates and bounce markers) with their extended labels, both under CC BY-NC-SA 4.0. The physics values tuned with this data and the plots generated from them (restitution coefficient plot) follow the same conditions. Raw data (tuning results, 24,300 rows of test hall judgments, actual benchmark results) is located in the working directory measure/tt-twin/ .

Table of Contents

• 1. Intuition: The Practice Arena with Known Answers
• 2. Real-World Tuning: 1,777 Actual Bounce Fragments
  • 2.1 Method
  • 2.2 Results
  • 2.3 However, Prediction Hardly Improved
  • 2.4 Most Errors Stem from Single-Camera Depth
• 3. The Referee Exam Hall with Known Answers: 24,300 Judgments
  • 3.1 What Was Built
  • 3.2 Three Weaknesses
• 4. One Loop of the Improvement Cycle: Trajectory Filling
  • 4.1 What Was Changed
  • 4.2 In the Simulator
  • 4.3 In Real Videos (Table Determining Adoption)
• 5. Expanding the AI Referee: Gravity Scale and Rule-by-Rule Judgment Table
  • 5.1 Gravity Scale
  • 5.2 Rule-by-Rule Judgment Table
• 6. Limitations
• 7. The Next Cycle
• References

1. Intuition: The Practice Arena with Known Answers

To improve the refereeing functions of a camera scoreboard, one must collect many "incorrect judgments." Actual match videos have few ground truth labels, and rare scenes like edge balls or obstructions almost never appear. Inside a simulator, physics knows where the ball landed, so desired scenes can be created in any quantity and shown to the referee. This is the exam hall with known answers.

However, if the ball in the exam hall flies differently from a real ball, the test will find the wrong weaknesses. Therefore, the procedure was divided into three steps.

1. Real-world tuning (real2sim): Tune the simulator's physics values using actual bounces.
2. Exam hall: Generate scenes using the tuned physics, feed them into the operational judgment code, and find weaknesses.
3. Improvement cycle: Fix one major weakness, and adopt the change only when it improves performance in boththe simulator and real videos.

2. Real-World Tuning: 1,777 Actual Bounce Fragments

2.1 Method

• Camera setup. The camera pose for each match was established using the four corners of the table tennis table. Since the four corners lie on a plane, the focal length cannot be determined and was fixed at 1,400 px (horizontal field of view approx. 69°), with ±20% sensitivity measured separately. Corner re-projection error was 1.0–4.0 px.
• Fitting bounce fragments. For each bounce, 11 unknowns were fitted, including contact time, contact point, incoming/outgoing velocity, and topspin. Constraints such as the contact point being constrained to the table plane and flight following gravity, drag, and Magnus effects determine the depth that is invisible with a single camera.
• Splitting. Tuning was split at the match level into 1,537 fragments from 5 training matches and 240 fragments from 7 test matches.
• Physics. The formulas were identical to the operational synthesis device (restitution e = e0 + e1·v_z, slip-roll boundary 2/5, drag and Magnus coefficients depending on velocity and rotation), and only restitution, friction, drag multiplier, and Magnus multiplier were solved as parameters.

Initially, tuning was done by minimizing the error of "predicting the next bounce by looking at only 50 ms after a bounce," but due to large measurement noise, the restitution coefficient drifted to unrealistic values (e0 1.09). This approach was discarded, and the process was changed to select drag and Magnus multipliers via a grid search and then directly regress the measured restitution coefficient for each bounce. Failed methods are also documented.

2.2 Results

The measured restitution coefficients were 0.04–0.07 lower than the values of the operational synthesis device. ITTF regulation (2.1.3) states that a ball dropped from 30 cm should bounce about 23 cm, and the values tuned from video shifted toward that standard (22.9 cm). This is in the same direction as MuJoCo's DeepMind table value (26.4 cm) and the operational default (26.0 cm) seen in Part 2, both of which were more elastic than regulations.

2.3 However, Prediction Hardly Improved

We predicted the remaining flight by looking at the fragment before the first bounce of a serve and 150 ms after the bounce, and compared it with the opponent's bounce (B2).

(Serves whose prediction hit the net and failed to produce B2 are excluded from error calculations. Training: 55–59, Test: 7–8.)

• The improvement in the 7 test matches is not significant. There are only 23 serves, and the 95% confidence intervals of the median (2,000 bootstrap iterations) overlap: position 11.3–30.9 cm vs. 12.7–29.4 cm, and time 6.6–32.9 ms vs. 8.9–38.3 ms.
• What tuning reduced was bias(training match time +9.7 → +3.1 ms), while the dispersion remained unchanged.

2.4 Most Errors Stem from Single-Camera Depth

The limits of the monocular camera were measured using re-sampled known answers. Treating 600 actual bounce fittings as ground truth, we re-rendered them with tuned physics, projected them onto the camera, added noise, and recovered them using the same method (acting as a lower bound assuming accurate physics models).

• Bounce locations can be determined well even with a single camera. It is 1–3 cm on a 60 fps phone.
• Depth-direction velocity is 7–10 times less accurate than longitudinal velocity. Over a 0.3-second flight, the depth position drifts by 10–50 cm, which matches the magnitude of dispersion in the table of Section 2.3.
• Rotation cannot be measured. The error is equal to or greater than the actual rotation magnitude (median of DeepMind data: 54 rad/s).
• Since 30 fps provides only 3 frames before and after, only 157 out of 600 could be fitted.
• Changing the assumed focal length to 1150 / 1400 / 1700 px moved the measured e (2.5–3.5 m/s column) to 0.847 / 0.831 / 0.802. The focal length assumption adds an uncertainty of about ±0.03 to the restitution coefficient.

Currently, depth errors overshadow differences in physics values. Determining the Magnus multiplier and depth simultaneously requires a second camera or high-speed video where ball markings are visible.

3. The Referee Exam Hall with Known Answers: 24,300 Judgments

3.1 What Was Built

• Physics. Table corners and sides were added to the values tuned in Section 2. The table was treated as a box with a panel thickness of 25 mm (estimated), and contact on the top surface edges from above was defined as an edge (according to ITTF 2.1.2, playing surface, point awarded).
• 9 Scenarios. Endline/sideline surroundings (edge, just inside, just outside), net-touch over, serve let, double bounce, fast smash, high lob, volley (hit by the receiver before bouncing; obstruction above the playing surface), and standard rally. Correct answers were determined by the contact order produced by physics rather than intent.
• 9 Degradation Conditions. Ideal (no noise), actual (measured detection noise), 30 fps, 120 fps, 2x noise, blur, occlusion, omission, and phone (30 fps + 2x noise + blur + occlusion). Detection confidence was sampled from the distribution of 71,617 operational detector frames.
• Judgment Engine. A copy of the operational judgment code was run with the same operational switches. 300 items per scenario across 9 conditions, totaling 24,300 judgments, took 25 minutes on a Mac.

Why Projection Coordinates Were Used Instead of Rendering. The input to the judgment engine is the screen coordinates of the ball, and the objective is to find the engine's weaknesses. At the coordinate level, tens of thousands of runs can be performed on the same scene simply by changing conditions. The trade-off is that differences in appearance are omittedThat is. The issues of the ball blending into the background, clothes, or lighting, misreading the racket or hand as the ball, and the actual blur shape were only mimicked using a detector error probability model. Actually, the baseline rally (actual condition) accuracy of the testbed, which is 82%, is lower than the 93% (54/58) of clean footage from 7 actual matches measured through the same operational pipeline. Therefore, the table below should be read only as a relative comparison between scenes and conditions. Absolute values should not be read as actual accuracy.

3.2 Three Weaknesses

1. Obstruction 16%(12–24% across all conditions). The engine does not have the concept that "the opponent's ball was hit above the playing surface before bouncing on my side," so it reads most as a double bounce and awards the point to the opposing player. The same engine gets 86% of volleys correct when the ball crosses the end line, but this is a coincidence, not because it understands the rules.
2. Edge ball 53%, incorrect without verification 21%. A ball touching the edge does not draw a V-shaped bounce on the screen. The engine fails to see the bounce and treats it as out. With the highest rate of being incorrect yet sent without human verification, this is the most dangerous cell.
3. Service let 60%(Occlusion 41%, phone 44%). It fails to see the speed reduction of a service grazing the net, misreads a ball deflected off the net as a bounce, or misses the bounce on the receiver's side.

Next was high lobs (in) at 60%. The ball goes off the top of the screen or flies for over 1.5 seconds, triggering "end of rally" prematurely. The number one cause of failure regardless of the scene was ball detection missing bounces in empty spaces.

120 fps performed worse than 60 fps(77% vs 80%). It might seem that more frames would be better, but this is believed to be because the engine's pixel threshold and bounce rejection rules are tuned for 60 and 30 fps. The causes were not isolated and re-measured (unmeasured). Confirming this through inference on actual 120 fps video is the next task.

4. One Loop of Improvement: Filling Trajectories

4.1 What Was Changed

We targeted the testbed's number one failure cause: "ball detection missing bounces in empty spaces for more than 4 samples (about 0.07 seconds at 60 fps)." Previous logs show that many missed bounces in actual matches also followed this path.

We fit a parabola with its curvature fixed to gravityto the trajectory segments before and after the empty space (curvature = screen scale at that position × 9.81, fitting only slope and intercept). The empty space is filled with the preceding and following curves only when all of the following conditions are met:

• The front goes down and the back goes up. The two curves meet within the empty space.
• The horizontal position of the intersection point matches within 4% of the table length from both sides, and the horizontal direction is maintained (reversal counts as a shot).
• The intersection point is within the table top stripe. The empty space is 0.3 seconds or less, the front and back segments are at least 3 samples each, and the parabolic residual is 2 px or less.

The filled point is assigned a confidence of half on both sides, and that bounce goes to human verification. The threshold was pre-determined by physics and was not chosen based on test results.

(The figure is a synthesized trajectory created to show the principle. It is not the actual decision result.)

4.2 In the Simulator

Incorrect judgments without verification decreased from 1,215 to 996. However, edges (53.1 → 49.9%) and obstructions (16.3 → 14.6%) decreased. This was not the goal of this change and remains a task for the next loop.

4.3 In Actual Footage (Table Determining Adoption)

Changes that improve performance only in the simulator are not adopted. Because the testbed is an approximation lacking appearance details, improvements there could worsen performance in actual footage. Therefore, we re-ran the fixed baseline table of actual footage using the exact operational real-time pipeline.

• No conditions worsened across the 12 test conditions (the −1 in dark occurred alongside +3 in the same condition).
• The challenging lob rallies were correctly changed. A high lob of 24.4 seconds in test match 1 was a rally where detection was empty at the moment the ball dropped, missing the bounce. It was corrected in clean, blur, camera shake, and phone quality conditions.
• There were five conditions in development where rallies worsened individually. Two rallies in clean footage involved a ball falling in front of the camera after the point ended, appearing above the table on screen and having its bounce filled. This is due to monocular depth ambiguity, and both decisions had a confidence of 0.44, sending them to human verification.
• Processing cost went from p95 0.445 to 0.545 ms per sample (clean training match 3), with the increased share being around 0.1 ms. This is well within the 5 ms real-time budget.

Decision: Adopted. However, it will be deployed with the switch turned off initially. When turned off, calculations remain the same as before so judgments do not change. Before turning it on, we will double-check "not worse than operation" in the real-time pipeline using the camera scoreboard's E2E logs (recording and replaying 7 test matches at 30 fps with a fake camera) and review the group of "balls falling in front of the camera" together.

5. Expanding the AI Referee: Gravity Ruler and Rule-by-Rule Judgment Table

5.1 Gravity Ruler

Service tosses must rise at least 16 cm after leaving the hand (ITTF 2.6.2). However, with a single camera, the distance to the ball is unknown, so there is no scale to convert screen pixels to centimeters. Here, gravity acts as a ruler. Fitting the screen height of the ball after leaving the hand to a parabola with respect to time y = a·t² + b·t + c yields a = g·s/2, where s is how many pixels correspond to 1 m at that ball's depth. Then, height can be measured as pixel difference ÷ s, and the 3D position of the ball can also be found via depth (focal length ÷ s).

• Actual measurement. The median scale measured this way across 80 service tosses in actual match footage was 330 px/m , and the net scale obtained from camera geometry was 303–335 px/m across four matches. The gravity ruler holds true in actual footage as well.
• Limitations. Rule 2.6.2 (16 cm toss) applies from the moment the ball leaves the hand. In synthetic tests where the hand lifts the ball by 3–15 cm before releasing it (estimated), the 16 cm judgment accuracy dropped to 0.70–0.86, and to 0.59–0.66 within the ±4 cm boundary. Under the condition of no hand lift, it was 0.99. The ball trajectory alone does not capture the "moment of release" well.
• In real footage (60 fps, 275 serves), the height could be measured in 80 serves (29%). Since there was no ground-truth height, accuracy could not be measured.

5.2 Judgment Table by Rule

The principle is to assign a confidence score to each signal, and if signals conflict or confidence is low, route it to human review. "Measured" refers to actual measurements from this test, and "Unmeasured" refers to unmeasured items.

6. Limitations

• Test environment accuracy is harsher than reality because appearance details are omitted (baseline rally 82% vs. actual 93%). Read the table for relative comparison only.
• Assumptions include table top thickness of 25 mm, sharp (non-rounded) corners, blur-miss model, noise intensity 2, and hand lift of 3–15 cm as estimates. Table side contact never occurred in scene generation.
• Camera focal length was assumed to be 1,400 px (e ±0.03 at ±20%), and spin cannot be determined monocularly.
• The before-and-after difference in serve prediction accuracy is not significant due to the test sample size of 23.
• The "incorrect without review" in the actual reference table is a proxy value counted using a rule with less than 0.6 confidence, rather than an operational calibration. Processing costs were measured under high Mac load, so they should be remeasured when idle.

7. Next Iteration

1. E2E check before turning on the trajectory interpolation switch.
2. Edge candidate → Human review. "Sharp deflection without V-shape" within the end-line band is marked as an edge candidate. First, measure whether incorrect answers without review decrease in the test environment and general outs do not worsen.
3. Obstruction rule. Since side views clearly show longitudinal position (Section 2.4), judge using the longitudinal coordinate of the direction-reversal point.
4. Test environment fidelity. Remeasure actual detector errors by speed and position to tune the noise model, and verify the result where 120 fps performed worse than 60 fps using actual 120 fps inference.

A word on TT. We tested the refereeing system 24,000 times in a training facility with known ground truth. We fixed the weaknesses found there and only accepted changes when performance also improved on real match footage. A referee that only does well in the training facility is useless!

References

All links were verified as accessible as of 2026-10-18.

• ITTF Table Tennis Regulations 2026–27 (Public Copy): https://www.tabletennisengland.co.uk/content/uploads/2026/07/Laws-of-Table-Tennis-2026-27.pdf
• DeepMind Public Ball State (used for spin magnitude comparison): https://github.com/google-deepmind/competitive_robot_table_tennis(© 2025 Google LLC, CC BY 4.0)
• Real video material: OpenTTGames dataset and its extended labels (CC BY-NC-SA 4.0). The matched physics values and restitution coefficient figures in this article follow the same conditions (CC BY-NC-SA 4.0).
• Previous part: Part 2: Reproducing Ping-Pong Ball Physics in MuJoCo, Part 6: Future Directions for the DeepMind Ping-Pong Robot