Pose and Racket: SimCC in RTMPose, Two Stages in MediaPipe, Five Points in RacketVision

Status: Draft (2026-10-13), partially reflecting actual measurements. Pose experiments (2026-10-11–12) and coaching screen speeds are measured values. For RacketVision, we only counted the data and did not run the model (unmeasured). Paper figures are marked as "Paper". Writing rule: Measured data is the default, and unmeasured items are marked as "Unmeasured".

Table of Contents

• 1. Why Look at People and Rackets
• 2. RTMPose: Coordinates as Two 1D Classifications
  • Top-down
  • SimCC: Two 1D Vectors Instead of a 2D Heatmap
  • Metrics
• 3. MediaPipe Pose: Separating Detection and Tracking
• 4. RacketVision: Five Points for the Racket
  • Counting the Table Tennis Part Manually (2026-10-13)
• 5. Where We Actually Used Them and Results
  • Score Engine: Deleting Hurts, Displaying Only Helps (2026-10-11–12)
  • Pose Coaching: MediaPipe for Browser, RTMPose for Server
• 6. Conclusion: What Goes Where
• References

1. Why Look at People and Rackets

As we will see in detail in Part 5, the biggest reason our current engine makes mistakes is event extraction. Among them, there were many mistakes in "reading a ball hit on the racket above the table as a bounce." If you only look at the ball's trajectory, both form a V-shape of "descending and then ascending" on the screen. The engine filters most of them out using the rule that "bounces maintain horizontal direction, while shots reverse it," but it cannot filter all of them.

Humans easily distinguish between the two. If the racket is next to the ball and the wrist has swung, it is a shot. Therefore, we intend to include pose (human joint coordinates) and racket position as auxiliary signals. There are three use cases.

And apart from scores, TT Lab's pose coaching screen uses the same model.

2. RTMPose: Coordinates as Two 1D Classifications

Top-down

RTMPose (arXiv 2303.07399, MMPose, Apache-2.0) is a two-stage method. First, a person detector (RTMDet) finds a person bounding box, and the keypoint model crops that box to find 17 joints. Even if a person appears small on one side of the screen, cropping and enlarging the box makes the joints clearer. This is robust to videos with fast, blurred arms and small people, like practice videos.

SimCC: Two 1D Vectors Instead of a 2D Heatmap

Ball detection in Part 2 drew 2D heatmaps. Drawing a 2D map for each of the 17 joints requires a lot of computation, and the map resolution directly dictates the position resolution. To make it precise, the map must be enlarged, which is costly.

SimCC (arXiv 2107.03332) changes the question. It solves "which bin is the x-coordinate of this joint?" and "which bin is the y-coordinate?" as separate classification problems. It outputs two 1D vectors of length $W \cdot k$ and $H \cdot k$ for each joint, and picks the highest bin in each vector. Here, $k$ is how many subdivisions a single pixel is split into (MMPose's simcc_split_ratio, default 2.0), and $k = 2$ yields half-pixel resolution. This effectively achieves sub-pixel precision without scaling up 2D heatmaps. This is the same idea as TTNet in Part 2 outputting ball positions as two x and y 1D vectors.

Decoding is very simple.

# open-mmlab/mmpose mmpose/codecs/utils/post_processing.py get_simcc_maximum() 에서(Apache-2.0)
x_locs = np.argmax(simcc_x, axis=1)
y_locs = np.argmax(simcc_y, axis=1)
locs = np.stack((x_locs, y_locs), axis=-1).astype(np.float32)
max_val_x = np.amax(simcc_x, axis=1)
max_val_y = np.amax(simcc_y, axis=1)
mask = max_val_x > max_val_y
max_val_x[mask] = max_val_y[mask]          # 신뢰도 = 두 축 최댓값 중 작은 쪽
vals = max_val_x
locs[vals <= 0.] = -1

Then keypoints /= simcc_split_ratio converts the bin number back to pixels. A notable point is that confidence is taken as the smaller oneof the two axes. If x is certain but y is ambiguous, that joint is considered ambiguous.

Metrics

RTMPose-m has 13.59 million parameters, 1.93 GFLOPs, and COCO AP of 74.9. Latency is 11.06ms on desktop CPU (ONNX Runtime) and 2.29ms on desktop-grade GPU (TensorRT FP16) (official README, 256×192 input). The 75.8 AP in the paper abstract is a value from a different training dataset version, so one must cite the correct version.

3. MediaPipe Pose: Separating Detection and Tracking

MediaPipe Pose Landmarker (BlazePose, arXiv 2006.10204, code Apache-2.0) is a model designed to run in mobile browsers. The idea is as follows.

1. Detector: Runs only initially or when the person is lost. It uses a face detector as a proxy for person detection and estimates the person's location by predicting the pelvis center and the radius and tilt of a circle enclosing the body.
2. Tracker: Runs every frame. It predicts 33 joints, "whether the person is still present," and the region of interest in the next frametogether. If it determines there is no person, the detector is run again in the next frame.

Not running heavy detection every frame is the secret to its speed. In addition to screen coordinates, it also provides 3D coordinates in meters with the pelvis as the origin(world landmarks).

License Warning: While the code is Apache-2.0, a separate model card is attached to the model file. We have not yet verified the terms of use in that card (unverified).

4. RacketVision: Five Points for the Racket

RacketVision (arXiv 2511.17045, AAAI 2026, code and data under MIT license) is a dataset and baseline model that provides ball locations and racket bounding boxes + five keypointsfor table tennis, tennis, and badminton broadcast videos. The five keypoints are top, bottom, handle, left, and right. The four points (top, bottom, left, right) define the outline of the racket face, while the handle determines the orientation. Knowing the viewing angle and orientation of the face allows us to ask, "Is the ball near the racket face?"

Direct Count of the Table Tennis Portion (2026-10-13)

We counted the read-only annotation files of the downloaded table tennis portion (refer to Part 7 for raw data and scripts).

Although they are called five keypoints, most rackets only show four. This is likely because the bottom end of the racket is obscured by the hand or overlaps with the face from the broadcast perspective. The paper also reported that the accuracy of the left and right keypoints (PCK@0.2 approx. 65–80%) is lower than that of the structural top and bottom keypoints (over 92%). Since a racket box is barely around 43px per side at 1080p, it is presumably even harder to find all five keypoints in side-angle phone videos.

Another result from the paper is crucial for the design. When predicting the future trajectory of the ball, simply concatenating racket features performed worse than using just the ball, and they only improved when combined via cross-attention (table tennis, past 80 frames → future 20 frames, ADE/FDE 113.9/184.3 → 101.3/161.3 px, from the paper). This leads to the same conclusion as our empirical measurement below: adding more signals does not automatically make things better.

5. Where We Actually Used It and the Results

Scoring Engine: Deleting Hurts, Displaying Helps (2026-10-11–12)

The ONNX file for the human detector (RTMDet) could not be read by OpenCV DNN due to the NMS operation inside the graph, and ONNX Runtime was not available in the experimental image. Therefore, instead of human bounding boxes, we put a 540×720 box centered on the ballinto RTMPose-m to ask only, "Is there a wrist near the ball?" With CPU (OpenCV DNN) taking 74ms per box (CPU shared with other tasks, median), we ran it only on the 2,499 frames where bounding candidates were generated by the engine.

First, we checked whether wrists could separate bounces in 5 training matches. Out of 1,982 bounce candidates, there were 1,481 true bounces, 27 false positives where a hit was read as a bounce, and 474 other false positives. Applying the rule "wrist score ≥ 0.5, wrist-to-ball distance ≤ 60px" eliminated 10 true positives, 5 hit false positives, and 24 other false positives. It barely distinguished true positives from false positives. This is partly because the engine had already filtered out most hit false positives, leaving only 27.

Here are the results from actual 12 matches and 275 rallies.

The conclusion can be summarized in two lines. Removingbounces with the current level of pose signals hurts performance . Simply highlighting them for human review without changing the decision brings 4 incorrect judgments without review in front of a human. Instead, "Needs Review" increases by 17 percentage points. Therefore, the engine adopted the rule: "Pose does not remove bounces (used only for Needs Review)." Runs with a properly integrated human detector are not yet measured.

Pose Coaching: MediaPipe for Browser, RTMPose for Server

6. Conclusion: What to Use Where

Models that take the ball, pose, and racket simultaneously to output "who hit it and when" are also emerging in table tennis (e.g., EventNet, which outputs racket contact timing using ball, table, and upper body keypoints, arXiv:2610.08286, achieving F1 91.16 within ±3 frames on the LATTE-MV broadcast dataset, paper). Testing such models on our side-view footage is a candidate for our next experiment (in progress).

TT Note: Just because the wrist is next to the ball, you shouldn't immediately shout "Hit!". In such cases, asking "Please take a look" was the way to protect the score.

References

• RTMPose: https://arxiv.org/abs/2303.07399, https://github.com/open-mmlab/mmpose (Apache-2.0)
• SimCC: https://arxiv.org/abs/2107.03332
• BlazePose: https://arxiv.org/abs/2006.10204, MediaPipe Pose Landmarker https://developers.google.com/edge/mediapipe/solutions/vision/pose_landmarker
• RacketVision: https://arxiv.org/abs/2511.17045, https://github.com/OrcustD/RacketVision (MIT)
• EventNet: https://arxiv.org/abs/2610.08286
• Our Measurements: Repository docs/tt/PERFORMANCE.md 2.9.5, Pose Coaching Design Document Section 9, docs/experiments/2026-10-13-tt-dataset-stats/