Status: Draft (2026-10-13). Paper figures were verified by opening the original texts along with their conditions. Hands-on testing was done using a numpy example on a laptop CPU (code/07_heatmap_decode.py, around 10 seconds). The actual results of using pose models in TT Score cite the measurements from Part 4 of the TT Score Model Anatomy. Series "Reading Vision Models 2", Part 7. Next part: Human Pose 2. The architecture of HRNet itself (keeping multiple resolutions side by side until the end) is covered in Reading Vision Models 1. This part focuses on "how keypoints are trained and extracted."

Human Pose 1: Finding Keypoints with Heatmaps

Table of Contents

• 1. Intuition: Maps Instead of Coordinates
• 2. Illustration: Two Paths for Handling Multiple People
• 3. OpenPose: Drawing Limb Directions as Maps
• 4. How to Train: Gaussian Targets and OKS
  • The Standard Top-Down Form: SimpleBaseline and HRNet
  • Evaluation: OKS and AP
• 5. Hands-on: Extracting Coordinates from Heatmaps
• 6. Paper Figures
• 7. Limitations
• 8. Connection to Our Project
• References

1. Intuition: Maps Instead of Coordinates

Pose estimation is the task of finding the positions of a person's joints (nose, shoulders, elbows, wrists, hips, knees, ankles, etc.) in an image. Based on the COCO dataset, there are 17 points per person.

The most straightforward method is to have the network directly output 34 coordinates (17 points × x, y). DeepPose did this in 2014. It regressed person-box-normalized coordinates using squared error and included a step to re-crop around the prediction and refine it once more (paper).

The same year, Tompson et al. proposed a different path. They had the network draw maps (heatmaps) shaped like the illustration for each keypoint, created ground truth as maps with small Gaussian hills at the keypoint locations, and compared them pixel by pixel using squared error. They also pointed out that direct coordinate regression is weak in very precise domains (paper). For nearly a decade since then, heatmaps have been the foundation of pose estimation.

The reason heatmaps are powerful is the same as what was explained for ball tracking in TT Score Model Anatomy Part 2. Training signals are generated per pixel, "this is not the place" is naturally expressed, and multiple candidates are allowed. Human pose has one more advantage. Since convolutions preserve the positional relationships in the image, outputs that represent positions as positionsfit the network architecture well.

2. Illustration: Two Paths for Handling Multiple People

When there are multiple people on the screen, another problem arises: "Whose wrist is this?"

• Top-down: A person detector first finds person bounding boxes, and each box is cropped, scaled up, and fed into a single-person pose model. Even if a person appears small, scaling them up ensures accuracy. However, processing time is proportional to the number of people, and anyone missed by the detector is lost for good.
• Bottom-up: Heatmaps for all keypoints are drawn across the entire image at once, along with cuesto group keypoints by person. Since the network runs only once, processing time remains nearly constant regardless of the number of people. However, the grouping step can fail, and it struggles with small people.

3. OpenPose: Drawing Limb Directions as Maps

The grouping cue in OpenPose (Cao et al., CVPR 2017) is PAF (Part Affinity Fields) . It draws a 2-channel map for each limb type, such as forearms. Pixels on the forearm receive a unit vector pointing "from the elbow to the wrist," while other pixels receive 0. When people overlap, vectors are averaged.

During testing, grouping works as follows:

1. Elbow candidates and wrist candidates are extracted from the peaks of the keypoint heatmaps.
2. For each line segment connecting an elbow and a wrist, scores are computed by integrating "PAF vector · segment direction" along the segment. For limbs belonging to the same person, the PAF aligns with the segment direction, resulting in a high score.
3. Since solving the entire problem at once is too difficult (NP-hard), it is solved separately for each limb type using bipartite matching(Hungarian algorithm) and then stitched together (paper).

Training is straightforward. Both keypoint heatmaps and PAF branches are output iteratively across multiple stages, and at each stage squared error against the ground truth maps is applied (intermediate supervision). Areas with unlabeled people are masked out in the loss function, because we must not teach the network that "there are no joints" simply because a label is missing.

The speed aspect is also interesting. For a video with 19 people, it achieved 8.8 fps on a laptop GPU (368×654 input), and with 9 people, the grouping step took 0.58 ms while the network took 99.6 ms (paper). The network runs once and grouping is very cheap, clearly demonstrating the advantage of bottom-up approaches.

4. How to Train: Gaussian Targets and OKS

The Standard Top-Down Form: SimpleBaseline and HRNet

SimpleBaseline (2018) is simple as its name implies. It attaches three deconvolution layers after ResNet to produce a 64×48 heatmap (¼ resolution) from a 256×192 input. HRNet (2019) avoids reducing and then increasing resolution; instead, it maintains high-resolution branches all the way through while continuously fusing multiple resolution branches (architecture covered in Series 1).

Training is identical for both.

정답 히트맵 Hⱼ(p) = exp(−‖p − 관절 j 의 위치‖² / (2σ²))     (히트맵 칸 단위)
L = Σⱼ Σ_p (예측 Ĥⱼ(p) − Hⱼ(p))²                              (화소마다 제곱 오차)

The HRNet paper text specifies σ as 1 pixel (the configuration value in the official code was not verified). Augmentation included rotation, 0.65–1.35x scaling, left-right flipping (left and right joint labels must also be swapped), and retaining only the upper or lower body (paper).

Evaluation: OKS and AP

Pose estimation requires a criterion to determine whether a prediction is "correct or incorrect." COCO uses OKS (Object Keypoint Similarity) .

OKS = 관절들에 대한 평균 exp( −d² / (2 · s² · (2σᵢ)²) )
      d = 예측과 정답의 거리, s² = 사람의 면적, σᵢ = 관절마다 정한 상수

σᵢ is a value determined by how consistently people annotate that particular keypoint. Joints with clear locations like wrists (0.062) are small, while ambiguous joints obscured by clothing like hips (0.107) are large. Even with the same distance error, wrists are penalized more heavily. AP is the average precision measured by varying the OKS threshold from 0.50 to 0.95.

5. Hands-on: Extracting Coordinates from Heatmaps

When drawing heatmaps at ¼ resolution, a single grid cell corresponds to 4 input pixels. Selecting the brightest cell (argmax) alone introduces an error of up to over 2 pixels. Two conventions exist to mitigate this:

• Quarter-shift: From the brightest cell, shift 0.25 cells toward the second brightest neighbor. SimpleBaseline and HRNet do this (paper).
• DARK (2020): After smoothing the predicted heatmap with a Gaussian, second-order Taylor expansion is applied to the log values around the maximum to find the peak with sub-pixel precision. Because the log of a Gaussian is a quadratic equation, this calculation fits perfectly. They also proposed that when drawing ground truth, hills should be placed at the sub-pixel locations of keypoints (avoiding quantization).

code/07_heatmap_decode.py , we compared the three methods. Placing the true position randomly down to decimal places 3,000 times, we decoded predictions generated by mixing perfect predictions (with Gaussians drawn at those spots) and noise (standard deviation 0.05) (average error in input pixels, code/07_heatmap_decode.out.txt).

Even with the same heatmap, simply changing the decoding method changes the error from 1.5 pixels down to 0.2 pixels . The coarser the grid (8 pixels), the larger the difference. DARK's 0.00 is because the target is a perfect Gaussian, making the Taylor expansion fit exactly; actual network outputs are not purely Gaussian, so real-world results are not quite that good.

We converted this error into OKS to see how critical it is for scores (assuming person area is half of a 256×192 box).

오차  2화소: 손목 0.995 · 엉덩이 0.998
오차  4화소: 손목 0.979 · 엉덩이 0.993
오차  8화소: 손목 0.919 · 엉덩이 0.972

At the OKS 0.95 threshold (the strictest AP tier), a wrist error of 4 pixels versus 8 pixels determines whether it is counted as "correct" or "incorrect." This is why a decoding difference of a few pixels translates to several points in AP.

6. Paper Figures

All values are based on COCO's 17 human keypoints and OKS-based AP. Since top-down methods yield results depending on the quality of person bounding boxes, the detector's AP is also listed.

Two observations stand out:

• Smaller inputs make decoding more critical. DARK gave +3.8 points at 128×96 and +1.2 points at 256×192. This is the same phenomenon seen in the example above where coarser grids led to larger differences. Furthermore, quarter-shift alone yielded an improvement from 61.2 to 66.9 (+5.7 points).
• Top-down approaches are generally more accurate. There is a significant gap between bottom-up (61.8–70.5) and top-down (73.7–77.0) methods from the same period. However, as we will see in Part 8, speed and multi-person trade-offs differ.

7. Limitations

• Heatmaps are expensive. It draws a 2D map for each joint, and precision is tied to the map's resolution. Increasing the resolution squares the computational cost. Part 8's SimCC addresses this issue.
• Fast arms blur. The wrist on the racket side during a table tennis swing blurs during the exposure time, so it is not a "single point." Methods like BlurBall that handle motion blur are not common for pose estimation (compare with Part 9).
• Occlusion. The opposite arm blocked by the torso and legs blocked by the table tennis table are frequently incorrect. COCO labels mark occluded joints as "present but invisible," but they are still difficult to predict correctly.

8. Connection to Our Project

• Actual measurement: The current level of pose signals could not be used to change decisions. In the TT Score experiment, when we fed RTMPose-m with a box centered on the ball instead of a person box and filtered bounce candidates using "is the wrist near the ball," it could hardly distinguish between correct and fake candidates; filtering them dropped the rally winner accuracy from 93.1% to 92.0% (60px) and 87.6% (90px). Using it only as a "requires review" flag reduced incorrect decisions without review from 10 to 6, but increased the "requires review" cases by 17 percentage points (actual data: Part 12, 275 rallies, TT Score Model Anatomy Part 4).
• Top-down for coaching. For pose coaching in practice videos, top-down is appropriate because there is one main subject. Server analysis uses RTMDet + RTMPose (top-down), and the browser uses MediaPipe (same Part 4).
• Two to four players for matches. It is two players for singles and four for doubles. Since the number of people is small, the cost of top-down is not significant, but there is the burden of running a separate detector. Single-stage methods like RTMO in Part 8 are candidates for this role.

TT a word. If you simply pick the brightest cell in the heatmap, when the cell is 4 pixels, it is off by an average of about 1.5 pixels. A small trick—"one-quarter of a cell toward the second brightest one"—cuts the error in half, which was an AP gain of 5 points. This is an area where small tricks make a big difference.

References

• Toshev & Szegedy, DeepPose: Human Pose Estimation via Deep Neural Networks, CVPR 2014. https://arxiv.org/abs/1312.4659
• Tompson et al., Joint Training of a Convolutional Network and a Graphical Model for Human Pose Estimation, NeurIPS 2014. https://arxiv.org/abs/1406.2984
• Cao et al., Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields(OpenPose), CVPR 2017. https://arxiv.org/abs/1611.08050 — Journal edition https://arxiv.org/abs/1812.08008
• Newell et al., Associative Embedding: End-to-End Learning for Joint Detection and Grouping, NeurIPS 2017. https://arxiv.org/abs/1611.05424
• Xiao, Wu, Wei, Simple Baselines for Human Pose Estimation and Tracking, ECCV 2018. https://arxiv.org/abs/1804.06208
• Sun et al., Deep High-Resolution Representation Learning for Human Pose Estimation(HRNet), CVPR 2019. https://arxiv.org/abs/1902.09212
• Zhang et al., Distribution-Aware Coordinate Representation for Human Pose Estimation(DARK), CVPR 2020. https://arxiv.org/abs/1910.06278
• Cheng et al., HigherHRNet: Scale-Aware Representation Learning for Bottom-Up Human Pose Estimation, CVPR 2020. https://arxiv.org/abs/1908.10357
• Example code and output: code/07_heatmap_decode.py, code/07_heatmap_decode.out.txt