The Learning Principles of Small and Fast Ball Tracking: What and How TrackNet V1–V5, WASB, TOTNet, and BlurBall Taught

Status: Draft (2026-10-13). Paper figures were verified with the original texts along with their conditions, while TrackNetV2 and V3 were verified through abstracts, official repositories, and descriptions in follow-up papers since the main texts could not be accessed (indicated as such). The hands-on part is a numpy example on a laptop CPU (code/09_ball_labels.py). Ball detection accuracy and speed measured on our videos are in TT Score Model Anatomy Part 2. This part does not repeat those measurements and covers only 「learning principles and lineage」. Added on 2026-10-15: TOTNet (public table tennis weights) and TrackNetV3 (RacketVision weights), which were previously left unmeasured, were re-tested under the same conditions, and the results are summarized in Section 8 (detailed tables in Part 2). The 「unconfirmed」 status of TrackNetV2 and V3 has also been revisited and corrected. Part 9 of the series 「Reading Vision Models 2」. From here on, it is about 「sports」.

Table of Contents

• 1. What Makes It Difficult: Three Imbalances
• 2. Pixel Imbalance: How the Loss Has Evolved
  • Direct Calculation: The Temptation of Lazy Prediction
  • TrackNet (2019): 256-Level Classification
  • TrackNetV2 (2020) and V3 (2023): Focal Loss-Style Weighted BCE
  • WASB (2023): Soft Labels Only for Hard Samples
• 3. Temporal Imbalance: How Frames Are Presented
• 4. Conditional Imbalance 1: Occlusion (TOTNet)
• 5. Conditional Imbalance 2: Motion Blur (BlurBall)
• 6. Paper Figures
• 7. Limitations
• 8. Connection to Our Project
• References

1. What Makes It Difficult: Three Imbalances

Since TT Score Part 2 describes 「why it is difficult」 in detail, we will only cover the main points. A ping-pong ball is about 3 pixels in diameter in a frame resized to 512×288, blurs to a length several times its diameter during the exposure time, and is frequently hidden by hands and rackets. From a learning perspective, this translates to three imbalances.

1. Pixel Imbalance: Almost every cell in the heatmap is 「not a ball」.
2. Conditional Imbalance: Most training data features clearly visible balls, while occluded, blurred, or stationary balls are rare. Yet, errors occur in those rare cases.
3. Temporal Imbalance: The ball position changes in every frame, but looking at just a single frame makes it impossible to distinguish between the ball and a fluorescent light.

The lineage below can be read as a history of addressing these three issues one by one.

2. Pixel Imbalance: How the Loss Has Evolved

Direct Calculation: The Temptation of Lazy Prediction

code/09_ball_labels.py We drew a ball disk with a radius of 2.5 cells on a 512×288 heatmap (the label size of WASB and TrackNetV3).

[1] 공 원판(반지름 2.5) 칸 수 20 / 전체 147,456 = 0.0136%

Out of 10,000 cells, 1.4 cells represent the ball. In such a problem, the very first thing a model learns is to answer 「there is no ball anywhere」. That alone gets 99.99% of the cells right. We compared the loss of a lazy prediction answering 「all 0.01」 with a prediction coloring the ball location with 0.6.

                 게으른 예측   공을 맞힌 예측   차이(학습 신호)   차이 / 게으른 손실
    BCE           0.010674      0.010118       0.000555         5.2%
    WBCE          0.000613      0.000012       0.000601         98.0%
    게으른 예측의 손실 가운데 「공 아닌 칸」 몫: BCE 94.1%, WBCE 0.2%

In standard Binary Cross-Entropy (BCE), 94% of the lazy prediction loss comes from accumulated small losses of 「non-ball cells」 that were already predicted correctly. Even when the ball is detected correctly, the loss decreases by only 5%. The learning signal gets buried in background noise.

TrackNet (2019): 256-Level Classification

The original TrackNet was trained via 256-class classification, choosing one of 0–255 brightness levels for each cell in the heatmap. The ground truth is a map scaling up a 2D Gaussian (σ² = 10) drawn at the ball center to 0–255 (paper). It takes 3 frames as input and outputs 1 heatmap for the last frame.

TrackNetV2 (2020) and V3 (2023): Focal Loss-Style Weighted BCE

The loss changes starting from V2. The formula confirmed in subsequent papers and the V3 official code is as follows (the V2 paper text could not be accessed).

WBCE = −[ (1 − p)² · y · log p  +  p² · (1 − y) · log(1 − p) ]

For cells already predicted correctly (where p is close to 0 for non-ball cells), p² is multiplied, making the loss nearly 0. Reducing the voice of easy cells is the core concept, which shares the same form as the detection-field focal loss (γ=2). In the calculation above, WBCE reduced the share of 「non-ball cells」 from 94% to 0.2%, and reduced the loss by 98% when the ball was correctly detected. The same difference (0.0006) now drives the entire loss.

V2 also changed to take 3 frames and output 3 frames at once(3-in-3-out), significantly increasing throughput (abstract: 2.6 → 31.8 FPS). V3 additionally inputs a background (the median of the frames) pre-computed for each match to make it easier to find 「what differs from the background」, and drew the ground truth as a binary disk with a radius of 2.5 (official code). Segments where the ball is missed are filled using a small network (InpaintNet) trained separately to interpolate after intentionally masking trajectory coordinates.

WASB (2023): Soft Labels Only for Hard Samples

WASB uses a small HRNet (a network preserving ball resolution all the way) with a quality focal loss. It initially trains with binary disk ground truths, and starting from epoch 20, selects only the training samples with poorly matched positions and changes their ground truths to real-valued Gaussians (minimum edge value 0.7) to continue training (HLSM, paper). In effect, well-predicted samples are taught with sharp ground truths, while difficult samples are taught with generous ground truths meaning 「roughly around here」. It was trained for 30 epochs from scratch without pre-training.

During inference, the heatmap is thresholded at 0.5 to find blobs, and the center weighted by heatmap values is used as the position. A simple online tracking method that discards candidates too far from positions predicted in previous frames was also attached (paper). Code for this post-processing is available in TT Score Part 2.

3. Temporal Imbalance: How Frames Are Presented

The motion maps of V4 and V5 are very low-cost approximations of the optical flow from Part 5. While optical flow calculates 「where and how much」, differences only look at 「where it changed」. This lineage argues that is quite sufficient for small balls. V5 applied a 10% dropout to the draft heatmap during training so that the refinement step does not simply copy the draft (paper).

4. Conditional Imbalance 1: Occlusion (TOTNet)

TOTNet (2025) directly targets occluded balls. Three things were changed in training (paper, some values from official repository).

• Artificially creating occlusion. Mimics occluded balls by covering the ball region in the target frame with the surrounding average color. Random regions in other frames are also covered to prevent the model from learning shortcuts to find the ball based solely on 「covered marks」.
• Varying ground truth and weights by visibility. Visible or partially occluded balls use single-pixel ground truths, completely occluded balls use spread Gaussian ground truths, and out-of-frame has no ground truth. Loss is BCE applied to x·y 1D distributions (same format as TTNet), multiplied by weights depending on the visibility level (repository example [1, 2, 2, 3]: heavier weight for heavier occlusion).
• 5-frame 3D convolution. Connects ball trajectories across preceding and succeeding frames to fill in occluded frames.

5. Conditional Imbalance 2: Motion Blur (BlurBall)

The core idea of BlurBall (2025, CVPRW 2026) lies in the labels.

• Placing position labels at the center of the blur. Not at the front edge of the blurred mark (the most recent position), but at the center. The center corresponds to the midpoint of the exposure time and is consistent. The half-length and angle of the blur were also labeled together (26 online videos, 64,119 frames, blur present in 62% of frames).
• Expanding the ground truth map across the entire blur segment. The ground truth is painted along the blurred segment rather than as a single dot at the ball center. WASB's loss and hard sample handling (HLSM) are kept as-is, while its soft ground truths are also expanded into segments.
• Blur length and angle are not regressed separately. They are extracted by performing Principal Component Analysis (PCA) on the blobs cut out from the heatmap (paper).

The latter part of the example uses a thought experiment to show why the 「center」 is good as a label. Given a ball speed of 20 m/s and an exposure time of 1/120 second (blur of about 17 cm), if a person is asked to look at a single frame and point to the 「front edge of the blur」, it becomes confusing which side is the front.

한 장만 보고 「끝점」 을 찍으면 앞뒤를 헷갈려 오차 표준편차 8.3cm, 「가운데」 는 방향과 무관해 0cm

While the assumption of being equally confused about front and back is extreme, the fact that the 「center」 remains the same position regardless of direction holds true regardless of the assumption.

6. Paper Figures

Since datasets, sports disciplines, and evaluation distances all differ, you should not compare across tables. Compare only within the same table.

TrackNet's random frame splitting 98.2 vs. match-separated cross-validation 84.3 is a warning for this entire field. Scores are inflated when neighboring frames from the same match are included in both training and testing. This is why the V4 paper pointed this out and created a new splitting protocol.

7. Limitations

• The evaluation distance of 4–5px is based on 512×288. Scaling up to 1080p leaves about 15 pixels, which is just a few centimeters on a table tennis table. Whether this is sufficient to determine "in or out" for a bounce requires separate consideration.
• Overlap between training and test videos. Since the list of 26 training videos for BlurBall is not public, it is unknown whether they overlap with OpenTTGames (2 videos in TT Score). Ball detection quality should not be interpreted as generalization performance.
• Real-time vs. recorded footage. V3's trajectory rectification requires future frames. The public code for TOTNet defaults to a 5-frame window, predicting the last frame to run without future frames (though a bidirectional setting also exists, which we confirmed matches the last frame in our measurements), but the computation is heavy. It took 259.5ms per window on an 8GB-class GPU.

8. Connection to Our Project

• Measurements are in Part 2. TT Score uses BlurBall as its base and directly measured precision 0.982 and recall 0.979 on 7 OpenTTGames test matches, weaknesses (stationary ball 0.790, occluded ball 0.863), server GPU 3.4ms/frame, and performance drops outside OpenTTGames. Results under degraded conditions compared to models like WASB and TrackNetV2 are also in that part (TT Score Model Anatomy, Part 2).
• Occlusion remains a weakness. The occluded ball score of 0.863 is precisely the condition targeted by TOTNet. Therefore, we ran the table tennis (TTA) weights released on 2026-10-15 directly on the same test (7 OpenTTGames test matches, 3,483 frames, 5px), and the recall for balls overlapping with players was 0.403 , less than half of BlurBall's, and the overall recall was 0.686. The paper's OpenTTGames accuracy of 0.97 comes from a model trained on OpenTTGames, meaning the weights learned in another gym could not be transferred as-is. Conversely, TrackNetV3 (RacketVision weights) trained on three sports broadcasts achieved precision 0.983 and recall 0.981—matching BlurBall's level—without ever seeing OpenTTGames, and found nearly stationary balls even better at 0.956 (BlurBall 0.790). However, overlapping balls scored 0.823 (TT Score Model Anatomy, Part 2, Section 6). In this experiment, "what videos it was trained on" mattered more than "whether the training method targeted occlusion." Since the TTA training data is research-only, the conditions for using those weights must also be considered.
• Teacher candidates for distillation. Models like V3-style rectification looking at past and future frames, and TOTNet retrained on our own video, fit the role of the "recorded footage teacher" in Part 4.

A Word on TT. Because the ball takes up 0.01% of the screen, naive training teaches the model to simply say "there is no ball." A loss function that scales down easy cells fixes this habit. From then on, it is a battle of how more frequently and heavily to show rare cases like occlusions and blurs.

References

• Huang et al., TrackNet: A Deep Learning Network for Tracking High-speed and Tiny Objects in Sports Applications, AVSS 2019. https://arxiv.org/abs/1907.03698
• Voeikov et al., TTNet: Real-time temporal and spatial video analysis of table tennis, CVPR 2020 Workshops. https://arxiv.org/abs/2004.09927
• Sun et al., TrackNetV2: Efficient Shuttlecock Tracking Network, ICPAI 2020. https://doi.org/10.1109/ICPAI51961.2020.00023
• Chen & Wang, TrackNetV3: Enhancing ShuttleCock Tracking with Augmentations and Trajectory Rectification, ACM MMAsia 2023. https://doi.org/10.1145/3595916.3626370 · Repository https://github.com/qaz812345/TrackNetV3
• Raj et al., TrackNetV4: Enhancing Fast Sports Object Tracking with Motion Attention Maps, ICASSP 2025. https://arxiv.org/abs/2409.14543
• Tang et al., TrackNetV5: Residual-Driven Spatio-Temporal Refinement and Motion Direction Decoupling for Fast Object Tracking, arXiv 2025. https://arxiv.org/abs/2512.02789
• Tarashima et al., Widely Applicable Strong Baseline for Sports Ball Detection and Tracking(WASB), BMVC 2023. https://arxiv.org/abs/2311.05237
• Xu et al., TOTNet: Occlusion-Aware Temporal Tracking for Robust Ball Detection in Sports Videos, CVIU 2026. https://arxiv.org/abs/2508.09650
• Gossard et al., BlurBall: Joint Ball and Motion Blur Estimation for Table Tennis Ball Tracking, CVPR 2026 Workshops. https://arxiv.org/abs/2509.18387
• Example code and output: code/09_ball_labels.py, code/09_ball_labels.out.txt