Skipping frames: Detection + tracking, keyframes, resolution, ROI — and 512×288 for ball tracking

Status: Draft (2026-10-16), partially measured. Latencies by input size for the ball model (BlurBall) were measured on a single 8GB laptop-class GPU in a home lab using disposable pods (median of 5 sets of 50 runs after 20 warmup runs). Skipping and ROI are synthetic trajectory simulations on a laptop CPU (code/05_skip_frames.py, around 1 second). Accuracy of skipping on real table tennis footage is unmeasured. Part 5 of the series "Reading Vision Models 3". The principles of optical flow are in Part 5 of Reading Vision Models 2, and the training of the ball tracking model is in Part 9of the same series.

Table of Contents

• 1. Intuition: Neighboring Frames Are Almost Identical
• 2. Rare Detection, Frequent Tracking
  • Try It Yourself: Skipping Frames with a Ping-Pong Ball
• 3. Keyframes: Reusing Features
• 4. Resolution: Reducing the Input
  • Our Measurements: Testing the Ball Model at Four Input Sizes
  • Choosing Size per Scene: AdaScale
• 5. ROI: Focusing Only Around Predicted Locations
• 6. Limitations
• 7. Connection to Our Project
• References

1. Intuition: Neighboring Frames Are Almost Identical

In a 30fps video, two neighboring frames are 33ms apart. In that time, the background remains the same and people move by a few pixels. However, running a heavy model from scratch every single frame repeats nearly identical calculations 30 times. All the methods in this post aim to reduce this redundancy.

2. Rare Detection, Frequent Tracking

SORT by Bewley et al. (2016) receives detection results every frame, predicts the next position for each object using a constant velocity Kalman filter, and matches predicted boxes with new detection boxes using IoU (Hungarian algorithm). The tracking part ran at 260Hz on a single CPU core around 2016 (Section 4.3, detection time appears to be excluded). Changing only the detector while keeping the same tracker raised the MOT accuracy (MOTA) from 15.1 to 34.0 (Table 1). This implies that tracking quality is ultimately determined by detection, while the tracker itself is almost free.

Taking advantage of this "almost free" tracker, we can perform detections infrequently . MediaPipe's pose tracking runs the detector only on the first frame and whenever the tracker loses the person, while for remaining frames it crops around the person using joint positions from the previous frame and runs only the joint model (official documentation). BlazePose (2020) achieved about 30 FPS with the Lite model on a single 2017 mobile CPU core using this approach (Table 1).

The prediction step of the Kalman filter is a single line.

상태 x = [위치, 속도],   다음 예측 x' = F·x,   F = [[1, Δt], [0, 1]]   (등속 가정)
검출이 오면 예측과 검출을 불확실성 비율로 섞어 고친다(갱신 단계)

Try It Yourself: Skipping Frames with a Ping-Pong Ball

The constant velocity assumption breaks down the moment a ball changes direction . Ping-pong balls hit the table or racket about once every 0.5 seconds. We created a synthetic rally (60fps, 1,161 frames, 29 bounces, 40 hits) on a 512×288 canvas with gravity, bounces, and hits, ran the detector once every k frames (position noise σ=1px), and filled the gaps using constant velocity prediction from the last two detections.

Looking only at the median, detecting every 3 frames seems fine at just over 2px. However, the tail (p95) reaches 46.8px before and after bounces and hits, which is more than nine times the ball diameter (approx. 5px at 512 width). The errors are concentrated precisely around the events we want to detect (bounces). This is why "infrequent detection," which works well for smoothly moving targets like human poses, is difficult to apply directly to ping-pong balls where scoring depends on bounces.

3. Keyframes: Reusing Features

Deep Feature Flow by Zhu et al. (2017) defines the skipped unit not as "detection results" but as "deep features." The heavy feature network (ResNet-101) is run only on keyframes, while intermediate frames warp the keyframe features along the flow computed by a lightweight optical flow network, followed by a lightweight head. They are trained end-to-end.

(A single 2013 datacenter GPU.) Accuracy dropped by 1–2 points while becoming 3.7–5x faster. Using an even lighter optical flow network reached up to ~10x (41.26 fps), but mAP dropped to 69.5 (Section 5.2). Since optical flow assumes "small movements over a single frame," it does not fit balls that fly several times their diameter in a single frame (table in Section 4 below).

4. Resolution: Reducing the Input

Computation is proportional to the number of pixels, i.e., width × height. Cutting the width in half reduces computation to one-fourth. In exchange, small objects become even smaller. Assuming a 2.74m ping-pong table width occupies 70% of the screen width, we calculated the size of the ball (40mm) and its movement during a single frame (ball speed 10m/s).

At any resolution, the ball moves 4.2 times (60fps) to 8.3 times (30fps) its diameter in a single frame. Therefore, the ball model observes multiple frames together rather than just one, using motion blur traces as clues (Part 9 of the same series 2).

Our Measurements: Testing the Ball Model at Four Input Sizes

We measured TT Score's ball model (BlurBall, 3-frame window, 1.49 million parameters) on an experimental GPU by changing only the input size. Since the model is fully convolutional, we can change the input size while keeping weights intact (training was done at 512×288, so accuracy at other sizes is unmeasured).

(Batch size 1, one window, FP16 uses autocast. Median of 250 runs after 20 warmup runs, and the medians of five sets matched within 0.1ms.)

Two observations can be made.

• Larger sizes slowed down faster than pixel scaling. 1024×576 has 4x the pixels but takes 5.45x the time. Because this model maintains high resolution throughout and carries feature maps with dozens of channels, larger inputs likely cause intermediate tensors to no longer fit in the GPU cache (reason estimated, profiling unmeasured).
• For smaller sizes, FP16 was actually slower(4.07 vs 4.69ms). When computation is so minimal that time is dominated by GPU waiting overhead (kernel launching, format conversion), reducing precision does not help. We will revisit this in Parts 3 and 6.

If allocating only inference to the 30fps budget (33ms), FP16 at 768×432 (24ms) fits. However, as seen in Part 1, the budget must also cover preprocessing, transmission, postprocessing, and other session shares, making 512×288 (FP16 9.8ms) a size that leaves sufficient margin. This value is also the resolution at which the ball model was trained.

Choosing Size per Scene: AdaScale

AdaScale by Chin et al. (2019) attaches a small regressor on top of detector features to predict the input size (short side 600, 480, 360, 240) to use for the next frame. Scenes containing only large objects can be fed at lower resolution, which can even reduce false positives and increase accuracy. On ImageNet VID, it achieved a 1.6x speedup while raising mAP from 74.2 to 75.5 (R-FCN, 2017 high-end consumer GPU, 75 → 47ms). The regressor itself takes about 2ms. While applying this idea directly to table tennis is difficult because the ball is always small, it can be used in forms such as "this scene is a break time with no visible ball, so make it small and sparse" (unmeasured).

5. ROI: Focusing Only Around Predicted Locations

If tracking predicts the next position, only a small window around it can be fed into the model. Instead of the whole screen, a 96×96 window accounts for 6% of the pixels. However, if predictions are off and the ball is outside the window, it gets missed, requiring the full screen to be re-evaluated. In the synthetic rally from Section 2, detection was run every frame (σ=1px) and window positions were predicted using the last two estimates.

Moments when the ball leaves the window are likewise concentrated at direction-changing moments. Since this simulation lacks occlusions, serve starts (ball suddenly appearing from a hand), and camera shakes, real-world misses would likely occur more frequently (unmeasured). Furthermore, models like BlurBall that process multiple frames together require cutting out the same location in preceding and following frames when cropping windows, so if the window follows the ball, the cue of "changes at the same location" gets blurred.

6. Limitations

• Sections 2 and 5 use synthetic trajectories. Real-world occlusions, motion blur, and cases where the ball goes out of frame and re-enters are not included.
• Measurements by input size in Section 4 timed execution only. We did not measure how accurately the ball model performs at sizes different from training.
• Paper figures come from individual GPUs and datasets, so multipliers cannot be directly compared against each other.

7. Connection to Our Project

• TT Score runs the ball model every frame. As shown in the table in Section 2, skipping frames causes errors around the bounce perimeter. Instead, we run the less expensive part "occasionally." Table detection runs once at the start of the match (EfficientTAM, Part 4), followed by optical flow tracking every frame, and redetecting only when severe shaking occurs. Running optical flow and cut determination every frame took about 1.7ms per frame (measured in the TT Score performance documentation, quiet node).
• The input size is 512×288. The phone downsizes and sends the camera feed at this resolution. As shown in the table in Section 4, a larger size quickly exceeds our budget, while a smaller size shrinks the ball below 3px. For a vertically held phone, matching this size reduces the screen width to 162px, making the ball almost disappear (5% ball visibility in the conditional tests of the performance documentation). This is why the shooting guide recommends landscape orientation.
• ROIs are not used yet. We first need to measure the cost of redetecting a lost ball versus the issue of multi-frame clues blurring when cropping windows (unmeasured).

The next post covers how batching affects latency and throughput when multiple users share a single GPU.

A word from TT. The ball travels four times its body length in a single frame and changes direction precisely at the moment it bounces. That is why I never close my eyes—instead, I only look at the table occasionally!

References

• Bewley, A. et al. (2016). Simple Online and Realtime Tracking. ICIP 2016. https://arxiv.org/abs/1602.00763
• Bazarevsky, V. et al. (2020). BlazePose: On-device Real-time Body Pose tracking. CVPR Workshop. https://arxiv.org/abs/2006.10204
• MediaPipe Pose documentation (Legacy solutions). https://github.com/google-ai-edge/mediapipe/blob/master/docs/solutions/pose.md
• Zhu, X. et al. (2017). Deep Feature Flow for Video Recognition. CVPR 2017. https://arxiv.org/abs/1611.07715
• Chin, T.-W., Ding, R., Marculescu, D. (2019). AdaScale: Towards Real-time Video Object Detection Using Adaptive Scaling. SysML 2019. https://arxiv.org/abs/1902.02910
• Our measurements: Actual measurements from this series (raw data in out-of-repo research logs), TT Score performance documentation sections 2.12 and 2.14
• Example code and output: In this folder, code/05_skip_frames.py, code/05_skip_frames.out.txt