U-Net and HRNet: Resolution-Preserving Architectures and Heatmap Output

Status: Draft (2026-10-13), partially measured. The examples in Sections 1 and 4 are values run directly on a laptop CPU (code/03_heatmap.py, 3 seconds). Paper figures are transcribed from the original texts along with their conditions. Measurements for the ball-tracking model are cited from TT Score Model Anatomy Part 2. Writing Rule: Measured data is the default; anything unmeasured is marked as "Unmeasured."

Table of Contents

• 1. Classification Shrinks, Location Must Be Preserved
• 2. U-Net (2015): Shrink, Enlarge, and Connect Corresponding Layers
  • Mechanisms for Training with Limited Data
• 3. HRNet (2019): Keeping High Resolution to the End
• 4. Heatmaps: Drawing Maps Instead of Coordinates
  • Try It Yourself: Decoding Heatmaps into Coordinates
• 5. Limitations
• 6. Connection to Our Project
• References

1. Classification Shrinks, Location Must Be Preserved

In Part 1, we mentioned that pooling broadens the receptive field while blurring locations. Let's see how much blurring actually occurs. We drew a 3px-diameter ball on a 32×32 image and applied repeated 2×2 average pooling (measured).

At 1/8 resolution, the ball is nearly indistinguishable from the background. This means that in the deep layers of classification models (ResNet typically shrinks down to 1/32), the precise location of the ball no longer remains. However, deep layers possess a broad context to determine "is this a ball?" An architecture that simultaneously possesses context (deep and wide view) and location (shallow and detailed view)is required. The answer branches into two directions.

2. U-Net (2015): Shrink, Enlarge, and Connect Corresponding Layers

Ronneberger et al. created U-Net for the segmentation of microscopic cell images (painting every pixel as either "cell" or "background"). True to its name, it has a U shape.

• Contracting Path (Encoder): Repeats two convolutions + 2×2 max pooling to reduce resolution and increase channels. Gathers context.
• Expanding Path (Decoder): Enlarges resolution back via upsampling.
• Skip Connection: Crops feature maps of the same resolution from the contracting path and concatenatesthem into the expanding path (concatenated along the channel dimension). The decoder paints by looking at both "deep-layer judgments" and "shallow-layer fine boundaries." nutty

ResNet's shortcut in Part 2 addsvalues, whereas U-Net's skip connection concatenatesthem. Addition slightly modifies existing information, while concatenation places different kinds of information (position and meaning) side by side for the next layer to blend.

Mechanisms for Training with Limited Data

Medical images have expensive labels. The training data in the paper consisted of about 30 electron microscopy segmentation images (512×512) and 35/20 cell tracking challenge images. Therefore, the design is tailored to data conservation.

• Unpadded convolutions. Boundaries are not filled with zeros, making the output smaller than the input (input 572×572 → output 388×388 in Figure 1). This is a choice to prevent learning fake border values. Large images are processed in overlapping tiles, and areas outside the borders are filled by mirroring the image (overlap-tile strategy).
• Elastic deformation augmentation. Each point of a 3×3 coarse grid is randomly displaced with a standard deviation of 10px and smoothly interpolated to warp the image. This is because cell tissues actually deform that way.
• Weighted loss. Assigns a large weight (w₀=10, σ≈5px) to the narrow background bands between touching cells to prevent painting two cells as one.

Results (Paper): Ranked 1st in the electron microscopy segmentation challenge with a warping error of 0.000353 (previous best 0.000420). IoU 0.9203(2nd place 0.83) and 0.7756(2nd place 0.46) on the two datasets of the 2015 Cell Tracking Challenge. Segmenting a 512×512 image took under a second on GPUs of that era, and training took 10 hours on a desktop GPU with 6GB of memory.

Beyond medical imaging, U-Net has become the baseline for almost any task requiring "producing a map of the same size as the input." TrackNetV2 for table tennis ball tracking also uses a U-Net architecture (TT Score Model Anatomy Part 2).

3. HRNet (2019): Keeping High Resolution to the End

U-Net shrinks once and then enlarges again. Although skip connections restore the positional information lost during shrinking, the high-resolution features received by the decoder belong to shallow layersand thus have weak semantic meaning. Sun et al. took a different path for human pose estimation (finding the locations of 17 joints). Maintaining a high-resolution stream from the beginning to the endis the core concept.

• Stage 1 starts with a single high-resolution (1/4 of input) stream.
• At each subsequent stage, a new stream with half the resolution and double the channels is added. Existing streams are kept as they are. The four stream channels of HRNet-W32 are 32, 64, 128, and 256.
• The streams are computed in parallel and exchangedat exchange units. Low-to-high resolution uses 1×1 convolution + nearest upsampling, while high-to-low uses 3×3 convolution with stride 2. This multi-resolution blending is repeated 8 times in total.
• The output heatmap comes from the highest-resolution stream.

As a result, the high-resolution stream continuously receives semantic information from deep layers without losing resolution.

(Paper Table 1. AP is the average joint location accuracy across multiple thresholds.) On the test-dev set, HRNet-W48 384×288 achieved 75.5. The main point of this table is that, with the same input, it is more accurate while having fewer parameters and computations than SimpleBaseline. "Not discarding resolution" proved to be more efficient than "discarding and recovering resolution."

4. Heatmap: Drawing a Map Instead of Coordinates

Both U-Net and HRNet output a map. For segmentation, it is a heatmap colored with the "probability of being an object" per pixel; for pose and ball detection, it is a heatmap colored with the probability of a joint (or ball) being presentper pixel. The ground truth is also created as a map. A small Gaussian hill is drawn at the joint location, and the Mean Squared Error (MSE) against the predicted map is measured per pixel. Stacked Hourglass (2016) used a hill with a standard deviation of 1px, the HRNet paper also mentions 1px, and the official configuration file uses σ=2. The output resolution is 1/4 of the input (256×192 → 64×48) based on the official HRNet configuration.

Trying It Out: Converting Heatmaps Back to Coordinates

Heatmaps ultimately need to be converted into coordinates to be useful. There are two methods.

1. argmax: The location of the brightest cell. It is simple, but an error up to the cell size remains.
2. Weighted centroid: A weighted average of the values of the cells exceeding a threshold (half of the maximum value). This restores the inter-cell position (sub-pixel accuracy).

Gaussian hills with σ=2 cells were drawn at 2,000 random locations in a 256×256 image, and the original coordinate errors were measured by decoding them at various heatmap strides (the reduction factor relative to the original) (measured values, ideal noise-free case).

Two things can be observed. First, the argmax error is directly proportional to the stride. Reading a 1/4-resolution heatmap with argmax simply loses an average of 1.5px. Second, the weighted centroid reduces the error by a factor of 7 to 8. Therefore, pose estimation code applies post-processing such as an argmax adjustment moving 1/4 of a cell toward the neighboring cell, and WASB for ball detection uses the weighted centroidof the blob. However, real heatmaps contain noise and secondary peaks (such as a fake ball from a fluorescent light), so they are not as clean as this table.

5. Limitations

• Memory. High-resolution feature maps consume memory proportional to the square of their size. Since HRNet's high-resolution branch remains active, batch sizes must be reduced for large inputs.
• Multiple Peaks. While heatmaps naturally express "multiple candidates," determining which one is truly correct must be handled separately. In ball detection, trajectories across frames serve this role.
• Occluded Objects. When two objects of the same class are close together, their hills merge. U-Net's "weighted loss between touching cells" was one solution to this problem.

6. Connection to Our Project

Ball detection in TT Score directly follows the lineage of this post (TT Score Model Anatomy Part 2).

• TrackNetV2 uses a U-Net architecture to take 3 consecutive frames and output 3 heatmaps.
• WASB and its table tennis counterpart, BlurBall , replaced the backbone with a small HRNet (approximately 1.5M parameters). The input is (B, 9, 288, 512)(RGB × 3 frames), and the weighted centroidof the connected blob thresholded at 0.5 is output as the ball position. This is the same method as the "weighted centroid" in Section 4.
• In the model input, the ball diameter is around 3px. As shown in the table in Section 1, at 1/8 resolution the ball disappears, making HRNet—which maintains the high-resolution branch all the way through—a great fit.
• RTMPose for pose estimation uses SimCC, which outputs two 1D distributions for the x and y axes instead of a heatmap. This is an alternative path that reduces computation by splitting a 2D map into two 1Ds (TT Score Model Anatomy Part 4).

The next post covers how these maps are scored and trained, detailing the loss functions and training techniques.

TT Note. Reducing to 1/8 makes the ball disappear. That is why the ball detection model keeps the large map all the way. It is heavy, but better than missing the ball.

References

• Ronneberger, O., Fischer, P., Brox, T. (2015). U-Net: Convolutional Networks for Biomedical Image Segmentation. MICCAI 2015. https://arxiv.org/abs/1505.04597
• Sun, K., Xiao, B., Liu, D., Wang, J. (2019). Deep High-Resolution Representation Learning for Human Pose Estimation. CVPR 2019. https://arxiv.org/abs/1902.09212
• Wang, J. et al. (2020). Deep High-Resolution Representation Learning for Visual Recognition. IEEE TPAMI. https://arxiv.org/abs/1908.07919
• Newell, A., Yang, K., Deng, J. (2016). Stacked Hourglass Networks for Human Pose Estimation. https://arxiv.org/abs/1603.06937
• Example code and output: in this folder code/03_heatmap.py, code/03_heatmap.out.txt