Loss Functions and Training Techniques: Cross-Entropy, Focal Loss, Heatmap Loss, Augmentation, Learning Rate Schedules, Mixed Precision

Status: Draft (2026-10-13), partially measured. Examples are values run directly on a notebook CPU(code/04_losses.py, under 1 second). Paper figures are transcribed from the original texts along with their conditions. Writing Rule: Measured data is the default, and unmeasured items are marked as "Unmeasured".

Table of Contents

• 1. Loss Functions are the "Grading Criteria"
• 2. Cross-Entropy
• 3. Focal Loss: Quiet Down Easy Examples
  • The Problem: 100,000 Background Cells
  • The Solution: Multiply by (1 − p_t)^γ
• 4. Heatmap Loss
• 5. Data Augmentation
• 6. Learning Rate Schedules and Large Batches
• 7. Mixed Precision
• 8. Limitations and Our Project
• References

1. Loss functions are "grading criteria"

As seen in Part 1, training is the process of adjusting weights to reduce loss. Therefore, what the model learns is determined by how the loss function is defined. If the grading criterion is the "number of incorrect answers," the student will try to get many easy questions right; if it is "extra points for difficult questions," they will obsess over hard ones. In detection, this difference actually becomes a major issue.

2. Cross Entropy

The standard loss for classification is Cross Entropy(CE). If the probability given by the model to the correct class is p_t , the loss is −log(p_t) . Giving 1 to the correct answer results in 0, and giving a value close to 0 causes it to approach infinity.

(Calculated directly.) Even examples that are already well-classified (p_t = 0.9) still have a loss of 0.1. While small for a single instance, the story changes when there are very many such examples.

3. Focal Loss: Quiet Down the Easy Examples

The Problem: 100,000 Background Cells

One-stage detectors (Part 6) determine whether an object is present for every pre-laid box (anchor) at every location in the image. The RetinaNet paper notes that there are about 100,000 such candidatesper image. Among them, only a few are actual objects, while the rest are sky, walls, and floors. Most of these are easy examples that the model quickly learns to correctly identify as "background."

I calculated this directly. It is the sum of losses for 100,000 background cells that are already well-classified ($p_t$ 0.95–0.999) and 100 object cells that are not yet classified ($p_t$ 0.05–0.5).

(Ground truth, code/04_losses.py [2].) With CE, 95% of the learning signal comes from "what is already known." The actual objects that need to be learned are drowned out by the noise.

The Solution: Multiply by $(1 - p_t)^\gamma$

The focal loss by Lin et al. multiplies CE by (1 − p_t)^γ .

FL(p_t) = −α_t · (1 − p_t)^γ · log(p_t)

For easy examples where $p_t$ is close to 1, (1 − p_t)^γ approaches 0, making the loss almost vanish, while hard examples remain largely unaffected. When $\gamma=2$, the loss for an example with $p_t=0.9$ is 1/100th of CE, and for $p_t=0.99$ it is 1/10,000th (the curve above matches the direct calculation and the paper's description).

Paper Metrics: $\gamma=2$ performed best and did not fluctuate wildly within the 0.5–5 range. When $\gamma=2$, $\alpha=0.25$ was optimal, though there was only a 0.4 AP difference from $\alpha=0.5$. The loss is applied across all ~100,000 anchors and divided by the number of anchors assigned to ground-truth objects. Trained with this loss, the one-stage detector RetinaNet (ResNet-101-FPN, 800px short side, scale-jitter augmentation, and 1.5x longer training) achieved 39.1 AP on COCO test-dev, surpassing the two-stage detectors of that time.

Highlighting the significance of the design, previously, easy backgrounds were discarded via sampling(selecting only hard negatives for training). Focal loss does not discard them; instead, it continuously down-weights them. It replaces data curation with a single line of loss function.

4. Heatmap Loss

What kind of loss is used to train the heatmap output from Part 3? There are two branches.

• MSE: Mean squared error per pixel between the ground-truth Gaussian hill and the predicted map. Used by Stacked Hourglass and HRNet.
• Penalty-reduced focal: Used by CornerNet (2018) for corner heatmaps and then by CenterNet (2019) for object center heatmaps. A focal-like loss is applied to the ground truth point (value 1), while the remaining pixels are multiplied by (1 − 정답값)^β so that pixels near the ground truth are penalized less. We use α=2, β=4 and divide by the number of ground truth points.

We measured the loss of several predictions for a σ=2 ground truth hill on a 64×64 heatmap (measured).

Looking at the MSE side, 「found nothing」 (0.00307) is lower than 「shifted by 4 pixels」 (0.00388). lower. Since the average is taken on a map where most pixels are 0, from the model's perspective, painting everything 0 incurs less loss than drawing the hill in the wrong place. This is why heatmap models tend to collapse into 「all zeros」 early in training. Because penalty-reduced focal divides by the number of ground truth points, the magnitude of the value is not overwhelmed by the number of pixels, and it also assigns a large loss (9.21) to 「all zeros」. However, in this example, the 4-pixel shift was penalized more heavily than 「all zeros」. This table shows that different losses penalize different things as worse.

5. Data Augmentation

Augmentation is showing the variations the model will actually encounter during training in advance. What to augment is essentially a declaration of 「variations the model should ignore」.

• AlexNet (2012): Random cropping + horizontal flipping effectively multiplied the training data by 2048, and color jitter reduced the top-1 error by over 1% (Part 1).
• U-Net (2015): Elastic deformation because cells actually bend that way (Part 3).
• mixup (2018): Mixes two images and two labels in the same ratio. x̃ = λx_i + (1−λ)x_j, ỹ = λy_i + (1−λ)y_j, where λ is sampled from Beta(α, α). Training an ImageNet ResNet-50 for 200 epochs reduced the top-1 error from 23.6% to 22.1% (α=0.2), but for 90 epochs, the gain was small, going from 23.5% to 23.3%. Augmentation is most effective when there is room for long training.
• The ViT family (DeiT) in Part 7 relies heavily on augmentation. Removing both mixup and CutMix dropped the ImageNet top-1 from 81.8% to 75.8%.

Augmentation must be changed alongside the labels. If you flip an image horizontally, you must also change the x-coordinate of the ball and the 「left/right hand」 tags of the pose. In table tennis, you must also be careful with the time axis. Feeding a ball-detection model with consecutive frames in reversed video order changes the ball's direction of motion.

6. Learning Rate Schedule and Large Batch

The learning rate is the step size of gradient descent. It is empirically known that changing the step size is better than using the same one from start to finish.

• Warmup: For the first few epochs, the learning rate is linearly increased from a small value to the target value. Weights initialized randomly have rough gradients, so a large step size from the beginning easily leads to divergence.
• Cosine annealing(SGDR, 2017): η_t = η_min + ½(η_max − η_min)(1 + cos(π·T_cur/T)). It decreases slowly at first, quickly in the middle, and slowly again at the end. The original paper proposed a scheme of increasing the learning rate again at each cycle (restart), but in practice, it is often used for a single cycle without restarts.

5 epochs of warmup + cosine, 100 epochs in total, with a maximum value of 0.1 (calculated directly).

에폭    0     2     4     5     25    50    75    99
lr    0.020 0.060 0.100 0.100 0.089 0.054 0.016 0.000

Linear scaling rule for large batches. Goyal et al. (2017) suggested that if the minibatch is multiplied by k, the learning rate should also be multiplied by klr = 0.1 · 배치/256. Instead, a warmup gradually increasing from 0.1 to 3.2 over the first 5 epochs was necessary. Using this, ResNet-50 was trained for 90 epochs in 1 hour with a batch size of 8192 and 256 datacenter GPUs, achieving an ImageNet validation top-1 error of 23.74% (±0.09), which was comparable to the batch 256 baseline of 23.60% (±0.12). Without warmup, it was 24.84%.

7. Mixed Precision

GPUs compute 16-bit floating point (half precision) much faster than 32-bit and use half the memory. However, simply switching to 16-bit breaks training. We checked this directly (measured).

FP16 가장 작은 정규수 6.10e-05, 가장 큰 수 65504
FP32 기울기           3.00e-04  2.00e-06  5.00e-08  1.00e-08
FP16 로 바로          3.00e-04  2.03e-06  5.96e-08  0.00e+00   ← 1e-8 이 0 이 되고 5e-8 은 6e-8 로 뭉개진다
×1024 → FP16 → ÷1024  3.00e-04  2.00e-06  5.00e-08  1.00e-08   ← 손실 스케일
FP16 가중치 1.0 에 1e-4 를 더하면 1.0                            ← 갱신이 사라진다

Two problems are visible. Very small gradients become 0 (underflow), and adding a small update to a large weight disappears due to rounding. Mixed precision training by Micikevicius et al. (2018) prevents this with three things.

1. FP32 master weights: Keep a 32-bit copy of the weights and perform updates there. Rounded to 16-bit only during computation (solution to the last line problem above).
2. Loss scaling: Before backpropagation, multiply the loss by a large number (between 8 and 32K in the paper, many networks do not need much) to pull the gradients into the range representable by FP16, and divide by it right before the update (third line above).
3. FP32 accumulation: Accumulate the results of 16-bit multiplications in 32-bit.

The paper reported that while weight memory increases by 50% due to the weight copies, activation values—which occupy most of the memory—become 16-bit, cutting the overall training memory by nearly half. Nowadays, bfloat16 , which has the same exponent range as FP32, is widely used, making underflow rare even without loss scaling. The speed measurement in the SAM 2 paper was also done using bfloat16 automatic mixed precision (Part 10).

8. Limitations and Our Project

• Loss functions faithfully learn the promise of the labels. BlurBall in TT Score was changed to place labels at the centerrather than the end of the blurred ball, and a model with the same architecture showed a significant difference in fast ball recall (TT Score Model Anatomy Part 2). Changing the label rules is just as much of a change to the 「scoring criteria」 as changing the loss function.
• Augmentation can simulate degradation conditions (darkness, low resolution, long exposure) in advance, but we have not yet retrained with such augmentations, and degradation conditions were used only for evaluation(Section 6 of the same post). Fine-tuning with mobile phone videos is in progress.
• Values like γ in focal loss, learning rate, and warmup length are values that need to be re-tuned for each problem. The values in the paper are merely a starting point.

From the next post, we cover detection. We start with two-stage detectors (R-CNN → Faster R-CNN) and anchors.

A word from TT. If 100,000 background grid cells all shout "That's the right background!" at once, the voice of a single ball gets drowned out. Focal loss tells the cells that were already classified correctly to quiet down.

References

• Lin, T.-Y., Goyal, P., Girshick, R., He, K., Dollár, P. (2017). Focal Loss for Dense Object Detection. ICCV 2017. https://arxiv.org/abs/1708.02002
• Law, H., Deng, J. (2018). CornerNet: Detecting Objects as Paired Keypoints. ECCV 2018. https://arxiv.org/abs/1808.01244
• Zhou, X., Wang, D., Krähenbühl, P. (2019). Objects as Points. arXiv. https://arxiv.org/abs/1904.07850
• Zhang, H., Cisse, M., Dauphin, Y. N., Lopez-Paz, D. (2018). mixup: Beyond Empirical Risk Minimization. ICLR 2018. https://arxiv.org/abs/1710.09412
• Loshchilov, I., Hutter, F. (2017). SGDR: Stochastic Gradient Descent with Warm Restarts. ICLR 2017. https://arxiv.org/abs/1608.03983
• Goyal, P. et al. (2017). Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour. arXiv. https://arxiv.org/abs/1706.02677
• Micikevicius, P. et al. (2018). Mixed Precision Training. ICLR 2018. https://arxiv.org/abs/1710.03740
• Touvron, H. et al. (2021). Training data-efficient image transformers & distillation through attention. ICML 2021. https://arxiv.org/abs/2012.12877
• Example code and output: In this folder, code/04_losses.py, code/04_losses.out.txt