Status: Draft (2026-10-13). Paper figures were verified by opening the original texts and checking their conditions. Practical experiments include a numpy example on a notebook CPU (code/08_simcc.py, under 1 second). For the actual results of using TT Score with RTMPose and MediaPipe, Part 4 of the TT Score Model Anatomy was cited, while ViTPose and RTMO were not run (unmeasured). Series "Reading Vision Models 2", Part 8. Previous part: Human Pose 1. Since RTMPose's decoding code and our use cases were covered in detail in TT Score Part 4, this part focuses on "why they are trained that way".

Human Pose 2: Between Real-Time and Accuracy

Table of Contents

• 1. Intuition: Scaling up one side, stripping down the other
• 2. ViTPose: Almost no decoder needed
• 3. SimCC: Turning coordinates into two classifications
  • Thoughts
  • Labels and Loss
  • Trying it out
  • Particularly strong at low resolution
• 4. RTMPose: Accumulating training recipes one by one
• 5. RTMO: Eliminating the person detector
• 6. Limitations
• 7. Connection to our project
• References

1. Intuition: Scaling up one side, stripping down the other

Looking at the table at the end of Part 7, the heavyweights of 2019–2020 used structures combining sophisticated networks like HRNet with elaborate decoding. Since then, paths have split into two.

• Accuracy side: Taking generalized transformers (ViT) that grew larger in the vision field as-is and relying on pre-training (MAE from Part 3) leaves almost no need for pose-specific devices (ViTPose).
• Speed side: Abandoning 2D heatmaps entirely, using cheaper outputs, and reclaiming accuracy by piling up training recipes (SimCC → RTMPose). Even eliminating the person detector (RTMO).

2. ViTPose: Almost no decoder needed

ViTPose (2022) uses an ordinary ViT as its backbone (1/16 resolution features in the figure) and places a very simple decoder on top. It produces a heatmap through a single sequence of "4x upsampling (bilinear interpolation) → ReLU → 3x3 convolution". Training uses the same heatmap mean squared error as Part 7.

The results comparing whether this simple decoder works across different backbones are interesting (COCO val). Changing to a simple decoder dropped the score by about 18 points on the ResNet backbone (ResNet-50 71.8 → 53.1), while it was within 0.3 points on the ViT backbone (ViT-B 75.8 → 75.5). Since the attention layers already mix a broad context, almost no work is left for the decoder(paper).

Accuracy follows the backbone size. For 256x192 inputs, ViTPose-B is 75.8, -L is 78.3, and -H is 79.1, and it reached COCO test-dev with a billion-parameter-scale backbone 80.9(single model, 576x432, external data added) (paper). Speed is also not as bad as one might think. ViTPose-B achieved 944 frames per second, and HRNet-W32 achieved 916 frames per second (datacenter-grade GPU, batch 64, paper).

Pre-training dataexperiments were also conducted. Performing MAE pre-training on 500,000 person crop images from COCO and AI Challenger instead of ImageNet still yielded 75.8, and using COCO alone yielded 74.5 (paper). This means pose data alone goes quite a long way, connecting to the "unlabeled our videos" discussion in Parts 2 and 3.

ViTPose also uses distillation from Part 4. Keeping the large model (L) frozen, only a single trainable "knowledge token" was trained, and that token was inserted into the small model (B). The token alone gave +0.2, and combined with output distillation it gave +0.8 (75.8 → 76.6) (paper).

3. SimCC: Turning coordinates into two classifications

Thoughts

Heatmaps had two problems (Part 7): they are expensive because a 2D map must be drawn for every joint, and precision is bound to the map resolution. SimCC (2022) changes the question. It solves "which bin is the x-coordinate in?" and "which bin is the y-coordinate in?" as separate classification problems.

• A single pixel is divided into k bins (division scale). If input width is 192 and k=2, there are 384 x-bins, and each bin is ½ pixel.
• Joint features are flattened and passed through a single linear layer to produce 384 x-logits and 512 y-logits.
• Decoding is done via argmax twice.

Finer resolution than pixels is achieved without enlarging the maps.

Labels and Loss

If you use a one-hot label where only the correct bin is 1, being one bin off or ten bins off are both equally "wrong". SimCC uses label smoothing to distribute probabilities around the correct answer, and uses the KL divergencebetween the predicted and target distributions as the loss. Comparing smoothing methods (ResNet-50, 384x288): no smoothing 71.3, uniform distribution 73.0, Gaussian 73.4, and Laplace 73.0 (paper). Gaussian, which considers closer bins less wrong, performed the best.

Trying it out

code/08_simcc.py output (code/08_simcc.out.txt).

[1] x 정답 101.3px → 칸 202.6/384, 정답 분포의 최대 칸 203, 폭(σ) 6.0칸
[2] τ=1.0: 예측 분포 최댓값 0.012, KL(정답‖예측) 1.420, 디코딩 105.5px
[2] τ=0.1: 예측 분포 최댓값 0.171, KL(정답‖예측) 0.499, 디코딩 105.5px

[2] demonstrates the role of temperature. Logits produced by the network usually have a narrow range of values. Applying softmax directly (τ=1) makes the distribution flat and far from the target distribution (KL 1.42), while dividing by τ=0.1 makes it sharp and closer (KL 0.50). RTMPose sets this temperature to 0.1, bringing the COCO val to 72.7 (step in the table below). Also, temperature does not change argmax(105.5px in both cases). Temperature is a knob that changes training gradients, not decoding results. The 4px error in this example is due to intentionally injected noise.

Quantization error was compared along a single x-axis (perfect predictions, 20,000 random locations).

When noise is mixed into the logits (1,000 times), the average errors for k=1, 2, and 3 become 2.97, 2.54, and 2.37px, making the benefit of finer bin splitting smaller as it gets buried in noise. The SimCC paper also tested k from 1 to 4 and stated that 2–3 were the best. Splitting too finely thins out the training signal per bin.

Output size (number of values per joint) is as follows.

2D 히트맵 ¼ 해상도 48×64 = 3,072  → 17관절 52,224
SimCC k=2: 384 + 512 = 896 → 17관절 15,232

Particularly strong at low resolution

SimCC's gains are larger when inputs are small (paper, COCO val, HRNet-W48).

This matches the conclusion of Part 7. The coarser the bins, the more problematic heatmap quantization becomes, and SimCC eliminates that quantization from the output structure.

4. RTMPose: Accumulating training recipes one by one

RTMPose (2023) is a top-down method. A person detector (RTMDet) finds boxes, and the CSPNeXt backbone → large convolution (7x7) → joint-specific features → attention layer (GAU) → two SimCC classifiers locate the joints. The value of this paper lies less in its structure than in the table documenting the effects of adding training recipes one by one(COCO val, RTMPose-m).

Labels take the form of "softmax applied to a distribution created by Gaussian distance", and temperature is applied to both prediction and label (paper). Heavy augmentations involved scale 0.6–1.4x, rotation 80 degrees, and regular application of Cutout. Since there are multiple versions with different training data conditions (COCO only / AI Challenger added / including flip test), citing numbers like 75.8 requires matching the specific version.

Speed(paper, RTMPose-m): Desktop CPU (ONNX Runtime) 11.06ms, desktop-grade GPU (TensorRT) 2.29ms, over 35fps on mobile SoC. The smaller version, RTMPose-s, achieves 13.89ms (over 70fps) at 72.2 AP on mobile SoC. The person detector is not run every frame, but every few frames, and boxes are generated using the previous frame's pose in between.

5. RTMO: Eliminating the person detector

The weakness of top-down approaches is that processing time scales with the number of people, along with the overhead of running a separate detector. RTMO (CVPR 2024), like object detectors (YOLO family), divides the image into a grid and predicts for each cell at once: "Is there a person here, and if so, what are their box and joints?"

Joint coordinates are produced as 1D classifications like SimCC, but instead of spanning the entire image, the bins are spread only inside the person box predicted by that cell (expanded by 1.25x) (Dynamic Coordinate Classifier, DCC). Whether a person is small or large, the inside of the box is divided into 192 x-bins and 256 y-bins, so the resolution adapts to the person's size.

The loss was also changed. Instead of KL divergence with the target distribution, it maximizes "the likelihood of the correct answer occurring under the Gaussian distribution around the predicted location", while having the network simultaneously learn the width of the distribution (uncertainty) (Maximum Likelihood Estimation, MLE). For the same RTMO-s, KL was 64.4, coordinate regression (OKS loss) was 65.6, and MLE was 67.6 (COCO val, paper). Much like SEA-RAFT in Part 5, this is an example where a loss that penalizes inherently ambiguous cases less ends up winning.

6. Limitations

• Accuracy and speed still involve a trade-off. There is a difference of nearly 9 points between ViTPose-G (80.9) and RTMPose-s (72.2). If you need to run it on a mobile phone, you have to choose from the lower end.
• COCO is not table tennis. COCO consists of everyday photos. Fast arms in a table tennis swing, blurred wrists, and legs blocked by the table are less represented. A paper's AP does not guarantee accuracy on table tennis videos (unmeasured).
• 2D joints alone do not reveal the racket face. It is hard to tell whether spin was applied or what the racket angle was from just a single wrist point. This is why attempts like RacketVision, which separately detects five racket points, emerged (TT Score Model Breakdown Part 4).

7. Connection to Our Project

• Actual measurement: RTMPose on the server, MediaPipe in the browser. Server analysis for TT Score coaching used RTMDet-m + RTMPose-m (ONNX Runtime). When sharing the server GPU, it took 50–98ms for person detection and 6–10ms/frame for joint detection. MediaPipe on the browser side took 12–15ms/frame in a headless browser on a laptop, and actual mobile phones were not measured (TT Score Part 4).
• Actual measurement: Heavy on the CPU. In scoring engine experiments, running RTMPose-m with OpenCV DNN on a CPU took 74ms for a single box (540×720) (median value sharing the CPU with other tasks, TT Score Part 4). This is why it cannot be included in every frame for 60fps real-time processing.
• Reasons to look at RTMO. In that experiment, OpenCV DNN could not read the NMS operation inside the ONNX file of the person detector RTMDet, forcing the use of a ball bounding box instead of a person box. RTMO, which lacks a separate detector, avoids this obstacle and handles four doubles players at once. It has not been received or benchmarked yet (unmeasured).
• A board grown through distillation. DWPose, which outputs 133 whole-body points, is a model that grew RTMPose through four stages of distillation. It is a candidate if finger joints become necessary.

TT's Note. By changing coordinates into a classification problem of "which cell?", the 3,072 cells used to draw the map were reduced to 896 cells. And I calculated and confirmed firsthand that temperature is not a knob for changing the correct answer, but a knob for changing how it learns.

References

• Xu et al., ViTPose: Simple Vision Transformer Baselines for Human Pose Estimation, NeurIPS 2022. https://arxiv.org/abs/2204.12484
• Li et al., SimCC: a Simple Coordinate Classification Perspective for Human Pose Estimation, ECCV 2022. https://arxiv.org/abs/2107.03332
• Jiang et al., RTMPose: Real-Time Multi-Person Pose Estimation based on MMPose, arXiv 2023 (Technical Report). https://arxiv.org/abs/2303.07399
• Lu et al., RTMO: Towards High-Performance One-Stage Real-Time Multi-Person Pose Estimation, CVPR 2024. https://arxiv.org/abs/2312.07526
• Yang et al., Effective Whole-body Pose Estimation with Two-stages Distillation(DWPose), ICCV 2023 Workshop. https://arxiv.org/abs/2307.15880
• Example code and output: code/08_simcc.py, code/08_simcc.out.txt