Reading 3D Trajectories and Spin from a Single Camera: TT3D, Uplifting Table Tennis, and TT4D — Physics-based vs. Learning-based Models

Status: Draft (2026-10-13). Paper figures were verified with the original texts and their conditions (Uplifting's conference confirmed via CVF open access: WACV 2026, pp. 7440–7449 — 2026-10-15). The hands-on experiment used a numpy example on a laptop CPU (code/11_spin_physics.py, taking around 30 seconds). We have never run this heuristic method on our own videos (unmeasured). Part of the series "Reading Vision Models 2," episode 11. Table corners and homography are covered in TT Score Model Anatomy episode 3, and 3D extremum rule planning in episode 5.

Table of Contents

• 1. Intuition: The Missing Dimension in 2D
• 2. Fitting with Physics Models: TT3D
• 3. Converters Trained on Synthetic Data: Uplifting Table Tennis
• 4. Lifting to 3D First, Then Splitting Events: TT4D
• 5. Hands-on: How Much Does Spin Curve the Ball and Can We Reverse-Fit It?
• 6. Paper Figures
• 7. Physics-based vs. Learning-based Models
• 8. Limitations
• 9. Connection to Our Project
• References

1. Intuition: The Missing Dimension in 2D

Cameras flatten the 3D world onto a 2D screen. In the process, depthis lost. A single point on the screen can be anywhere along the line of sight extending from the camera. The 2D position of a ball alone does not reveal whether it is 2 meters or 4 meters away from the camera.

However, table tennis offers many clues.

• The dimensions of the table are fixed(2.74m × 1.525m). If all four corners are visible, the camera's position, orientation, and focal length can be calculated.
• At the moment of a bounce, the ball is on the table. The intersection of the line of sight at that exact moment and the table plane gives the 3D position of the ball.
• The ball follows the laws of physics. The trajectory between bounces is largely determined by gravity, air resistance, and the force generated by spin (the Magnus force).

The final clue even reveals the spin. A ball with topspin drops faster, while underspin causes it to float further. By observing the curvature of the trajectory, the spin can be retrocalculated.

2. Fitting with Physics Models: TT3D

TT3D (CVPR 2025 Workshop) minimizes learning and relies heavily on physics.

1. Camera Calibration: A table segmentation model (a small U-Net++, 1,200 labeled images, Dice loss) detects the table region, and its four corners are used to simultaneously solve for the camera's position, orientation, and focal length (alternating between PnP and focal length search). A Kalman filter smooths these values over time. The corner error was 2.39 ± 1.47 pixels (evaluated on 40 images, from the paper).
2. 3D Position of Bounces: This is the intersection point between the line of sight at the bounce moment and the table plane. The bounce timestamp is found by fitting piecewise quadratic curves to the 2D trajectory and incorporating the blur angle computed by BlurBall (Episode 9).
3. Velocity and Spin: Treating velocity (3 components) and spin (3 components) right before a bounce as unknowns, the flight equations are integrated using these values to optimize the reprojected screen position so that it closely matches the observed 2D position (minimizing reprojection error).

The flight equations are as follows (from the paper, notation altered):

m · dv/dt = −k_D · |v| · v  +  k_M · (ω × v)  +  m · g
m = 2.7g,  k_D = 3.8 × 10⁻⁴,  k_M = 4.86 × 10⁻⁶,  공 반지름 2cm

The first term represents air resistance, the second term the Magnus force, and the third gravity. Bounces are modeled with a restitution coefficient of 0.85 and a friction coefficient of 0.3, calculating whether the ball slides or rolls on the table. (The paper used a positive sign for k_D, but to ensure air resistance opposes motion, sign conventions must be aligned. The TT4D appendix uses a minus sign for the same k_D, and defines k_M as 3 × 10⁻⁶, differing from TT3D. Our example uses the TT3D values with a negative sign.)

3. Converters Trained on Synthetic Data: Uplifting Table Tennis

While physics fitting is accurate, it is slow, and missing observations or high noise can trap the optimization in local minima. Two papers by Kienzle et al. take a different route. They train a transformer that directly takes a 2D point sequenceand outputs 3D point sequences and spin.

The challenge is ground truth. Real videos lack ground truth for both the 3D position of the ball and its spin. Therefore, ground truth is generated using a simulator.

• A physics simulator (MuJoCo) is used to generate a large volume of table tennis trajectories. The 2025 version uses 50,000 trajectories, while the 2026 version (Uplifting) uses 140,000 trajectories (approximately 4.7 million 2D points). These serve as "free ground truth" where both 3D positions and spin are fully known.
• These 3D trajectories are then projected into 2D using virtual cameras to create the inputs.
• Training is based on the mean squared error of the 3D trajectory plus the mean squared error of the initial spin (2025 version, from the paper).

The critical bridge from synthetic data to reality is augmentation. Because real detectors miss the ball and frame rates vary, 5% of points were randomly dropped during training, and frame rates were sampled between 20–60 fps. Sudden trajectory cutoffs, motion blur, and fuzziness were also simulated. Furthermore, instead of token indices, the transformer's positional encoding utilized actual timestamps(rounded to the nearest 2ms), allowing the model to understand temporal relationships even when frames are dropped or intervals vary (from the paper).

The frontend 2D detector is trained on real video. The ball is detected across 3 consecutive frames, and the table is found using 13 reference points via heatmaps (squared error, σ=6 pixels). The ball detector was pre-trained on 50,000 frames of BlurBall data and then fine-tuned on 1080p table tennis footage (TTHQ). If the results disagree with an auxiliary detector of a different architecture, the detection is discarded (from the paper).

4. Lifting to 3D First, Then Splitting Events: TT4D

TT4D (ACM MM 2026) reverses the order. While previous methods first detected bounces and hits in 2D trajectories and then lifted the segments in between to 3D, TT4D lifts the entire rally to 3D first and then segments the events.

• Bounce = local minimum of height z.
• Hit = extremum along the table's length axis x (where the ball reverses direction). Simple rules are added, such as requiring at least 0.2 seconds between hits and at least 0.3 meters from the table edge.

In 2D views, bounces and hits both look like "V-shaped bends" and are easily confused, but in 3D they are fundamentally defined as "touching the floor" versus "reversing direction." Training was performed on roughly 3 million synthetic trajectory points, with occluded frames masked out during training and filled using interpolation tokens. The model is lightweight with 1.6 million parameters (from the paper).

5. Hands-on: How Much Does Spin Curve the Ball and Can We Reverse-Fit It?

code/11_spin_physics.py integrates the flight equations above at 1ms intervals (code/11_spin_physics.out.txt).

Landing Spot Shift Caused by Spin. A ball hit from 30cm above the player's baseline at an initial speed of 7.05 m/s (at a slight upward angle of 6.5°).

아랫회전 50rps: x = 2.68m, 비행 463ms
회전 없음    : x = 2.09m, 비행 348ms
윗회전 50rps : x = 1.74m, 비행 284ms
참고: 공기가 없다면 x = 2.39m. 탁구대 길이는 2.74m

A spin of 50 revolutions per second shifts the initial landing point from 1.74m to 2.68m—a difference of nearly 1m. Flight times also vary significantly, from 284ms to 463ms. Even without spin, air resistance causes the ball to land 30cm short compared to a vacuum. Because the ball is lightweight (2.7g), air effects are substantial.

Reverse-Fitting. The flight segment before the first bounce of a ball with 50 rps topspin was observed at 60fps and 30fps with added 3D position noise, and 3 initial velocity components and 1 spin component were fitted using Gauss-Newton least squares (repeated 30 times). This is a heavily simplified version of the TT3D method. The spin axis was constrained to a single lateral axis.

When position noise is 1cm, the spin direction is always correct and the magnitude is within roughly ±6 rps. At 3cm, the magnitude becomes practically unreliable, and at 30fps even the direction is wrong 13% of the time. Because spin is concealed within microscopic curvatures of the trajectory, position errors of just a few centimeters completely mask the signal. In reality, only 2D positions—not 3D positions—are observed, making it even harder. Consequently, rather than regressing spin as an exact value in rps, papers tend to evaluate it via topspin/backspin classification(see table below).

6. Paper Figures

The final row requires extreme caution when citing. TT3D reports 8.9cm on its own benchmark, but scores 29.86cm on TT4D's benchmark. Different benchmarks can yield numbers that differ by more than a factor of three for the exact same method.

7. Physics-based vs. Learning-based Models

These two approaches are closer than you might think, because the "ground truth" for learning models ultimately comes from physics simulators. The difference is whether you solve the physics every time during testing (optimization) or internalize it once during training (augmented synthetic data).

8. Limitations

• Fixed camera assumption. All three methods assume the camera does not move during a rally. Handheld phones shake (while TT3D smooths corrections using a Kalman filter, large movements are a separate issue).
• Real 3D ground truth is rare. TT3D's multi-camera ground truth is an exception; the others rely on indirect evaluation via reprojection error or rotation classification.
• Licenses. The Uplifting code is under GPL-3.0, the TT3D repository lacks a license file, and TT4D has not been released yet (as of this research report). Our principle is never to import GPL code, but rather to implement the methods ourselves.

9. Connection to Our Project

• Current engine handles 2D and the table plane. TT Score computes a homography using the four corners of the table to map the 2D ball position to table-plane coordinates, detecting bounces from the inflection points of the 2D trajectory (TT Score Model Breakdown Parts 3 & 5). A large portion of the remaining errors stems from "false bounces interpreted from hits," which happens because both look like the same V-shape in 2D.
• Reviewing TT4D's equation-based 3D extremum rules. Moving to 3D separates the definitions of a bounce (z minimum) and a hit (x extremum). This was planned in Part 5 and has not yet been experimentally tested (unmeasured).
• Side-view filming is most advantageous. In TT3D's results, the side view was most accurate at 8.9 cm, while the back view was 22.3 cm. This aligns with our recommended filming position (side of the table, on a tripod).
• Spin as a "bonus." As seen in the examples, spin magnitude is extremely sensitive to positional noise. It is unnecessary for scoring, and it is realistic to use it merely as a classification for coaching like "was that topspin?" (estimation).

A word from TT. A ball spinning at 50 revolutions per second can land nearly a meter apart even when hit with the same force. Conversely, inferring spin from the trajectory required knowing the ball's position within 1 cm. Deflecting is large, reading is hard. That is spin for you.

References

• Gossard et al., TT3D: Table Tennis 3D Reconstruction, CVPR 2025 Workshops (CVSports). https://arxiv.org/abs/2504.10035
• Kienzle et al., Towards Ball Spin and Trajectory Analysis in Table Tennis Broadcast Videos via Physically Grounded Synthetic-to-Real Transfer, CVPR 2025 Workshops. https://arxiv.org/abs/2504.19863
• Kienzle et al., Uplifting Table Tennis: A Robust, Real-World Application for 3D Trajectory and Spin Estimation, WACV 2026, pp. 7440–7449 (CVF Open Access). https://arxiv.org/abs/2511.20250
• Rahmanian et al., TT4D: A Pipeline and Dataset for Table Tennis 4D Reconstruction From Monocular Videos, ACM MM 2026. https://arxiv.org/abs/2605.01234
• Example code and output: code/11_spin_physics.py, code/11_spin_physics.out.txt