Status: Draft (2026-10-13), partially measured. The experiments in Section 4 were run directly on a laptop CPU (Python 3.9, PyTorch 2.8) (
code/02_residual.py, about 7 minutes). Paper figures are transcribed from the original text along with their conditions. Writing rules: Measured data is the default, and unmeasured items are marked as "Unmeasured".
Table of Contents
- 1. The Degradation Paradox
- 2. Residual Connections: Learn Only the Residuals
- 3. Batch Normalization
- 4. Trying It Yourself: Plain vs. Residual, With and Without Normalization
- 5. Paper Figures
- 6. Limitations
- 7. Connection to Our Project
- References
1. The Degradation Paradox
AlexNet in Part 1 had 8 layers. Since stacking more layers allows the network to learn more complex patterns, performance should theoretically improve. However, in 2015, He et al. reported the opposite phenomenon. In a plain convolutional network (plain), the ImageNet validation top-1 error (average over 10 crops) was as follows:
| Network | 18 Layers | 34 Layers |
|---|---|---|
| Plain Network | 27.94% | 28.54%(Worse) |
| ResNet (Residual Connections) | 27.88% | 25.03% |
(Table 2 in the paper.) The crucial point is that the 34-layer plain network had a higher training error than the 18-layer network. If it performed worse only on the test set, it would be overfitting (memorization), but performing worse on the training data as well indicates an "optimization failure."
This is strange. A 34-layer network can emulate an 18-layer network by simply copying the 18 layers and letting the remaining 16 layers act as "identity mapping layers that pass the input through unchanged." Therefore, a deeper network should perform at least as well as a shallower one. Yet, gradient descent failed to find that solution. Adjusting weights so that multiple layers of non-linear transformations become precisely an identity function is harder than it seems.
2. Residual Connections: Learn Only the Residuals
The core idea of ResNet can be stated in a single sentence: Instead of having a block learn the H(x) entire output , have it learn only the F(x) = H(x) − x residual to be added to the input
그냥 쌓은 블록: y = F(x)
잔차 블록: y = x + F(x) (F = 합성곱 두세 층)
. F This makes learning the identity function easy. By simply initializing the weights of y = x near zero, we get . Even with deep stacking, we can start from a state where "unnecessary blocks do nothing." Because the shortcut (skip connection) is a single addition, it adds almost no parameters or computation time.
Gradient Flow
Examining what happens during backpropagation clarifies why this design works. In a plain network, the gradient of an earlier layer is obtained by multiplying the derivatives of the subsequent layers. If the multiplied values are slightly less than 1, the gradient vanishes to 0; if they are slightly greater, it explodes.
When $L$ residual blocks are stacked, the follow-up paper by the same authors (Identity Mappings, 2016) shows that it becomes:
x_L = x_l + Σ_{i=l..L-1} F(x_i)
∂E/∂x_l = ∂E/∂x_L · ( 1 + ∂/∂x_l Σ F(x_i) )
The 1 inside the parentheses is the key. The loss gradient ∂E/∂x_L is passed directly down to earlier layers without passing through any weights. Since the second term is unlikely to be exactly $-1$ across the entire mini-batch, the paper explains that gradients do not vanish no matter how small the weights are. The chain of multiplications has been replaced by a chain of additions.
This paper adopted a "pre-activation" order (normalization $ ightarrow$ ReLU $ ightarrow$ convolution) where the activation function comes after the addition, making the shortcut completely clean. This allowed them to successfully train a 1001-layer network on CIFAR-10, achieving an error rate of 4.62% (median of 5 runs with a mini-batch of 64; 4.92% with a mini-batch of 128). The original order ResNet of the same depth had a 7.61% error rate.
3. Batch Normalization
In ResNet blocks, every convolution is followed by Batch Normalization(BN). Within a mini-batch, the mean is subtracted and divided by the standard deviation for each channel, followed by scaling and shifting using two learnable parameters ($\gamma, \beta$).
x̂ = (x − 배치평균) / √(배치분산 + ε)
y = γ · x̂ + β
During inference, moving averages collected during training are used instead of batch statistics.
Explanation in the original paper (2015). Ioffe and Szegedy believed that during training, when the weights of earlier layers change, the distribution of inputs received by later layers continuously shifts ("internal covariate shift"), necessitating a small learning rate. They argued that fixing the distribution through normalization allows for a large learning rate. The effect was profound: Inception required 31 million steps to reach the same accuracy (72.2% validation), whereas the version with BN required only 2.1 million steps ("14x fewer steps" in the paper). An ensemble of 6 BN networks achieved an ImageNet validation top-5 error of 4.9% and a test error of 4.82%.
Explanation in later research (2018). Santurkar et al. argued that the effect of BN has little to do with distribution stability. Even when time-varying noise was artificially injected after BN layers to disturb the distribution, performance remained almost identical, and they presented measurements suggesting BN might not actually reduce internal covariate shift. Instead, they demonstrated through theory and experiment that BN smooths the loss landscape (improving the Lipschitz properties of loss and gradients), causing gradients to point in more reliable directions.
This is a great example of how the explanation for "why it works" changed over time. The practical conclusion remains the same: BN facilitates the training of deep networks with large learning rates. However, as shown in the experiment below, BN alone does not solve the depth problem.
4. Trying It Yourself: Plain vs. Residual, With and Without Normalization
This is a toy experiment using fully connected layers of width 32 instead of convolutions. A classification task separating 2D points into two spirals (512 points) was trained for 500 steps using SGD + momentum. For each network, the best learning rate among 0.01, 0.03, and 0.1 was chosen (since fixing a single learning rate would let the choice favor a specific network and dictate the conclusion), and the values represent the median of 3 seeds. The residual blocks initialized the second-layer weights to 0 so they acted as identity functions at the start.
| Network | Depth | Initial First-Layer Gradient | Final Loss | Training Accuracy |
|---|---|---|---|---|
| Plain | 4 | 3.1×10⁻² | 0.000 | 1.000 |
| Plain | 16 | 1.2×10⁻² | 0.028 | 0.996 |
| Plain | 64 | 9.5×10⁻³ | 0.692 | 0.520 |
| Residual | 4 | 3.2×10⁻² | 0.000 | 1.000 |
| Residual | 16 | 4.1×10⁻² | 0.577 | 0.855 |
| Residual | 64 | 3.5×10⁻² | 0.692 | 0.520 |
| Plain + BN | 4 | 3.4×10⁻¹ | 0.000 | 1.000 |
| Plain + BN | 16 | 1.4 | 0.000 | 1.000 |
| Plain + BN | 64 | 1.1×10⁴ | 0.489 | 0.588 |
| Residual + BN | 4 | 3.2×10⁻² | 0.000 | 1.000 |
| Residual + BN | 16 | 4.1×10⁻² | 0.000 | 1.000 |
| Residual + BN | 64 | 3.5×10⁻² | 0.000 | 1.000 |
(Measured 2026-10-13, laptop CPU. An accuracy of 0.5 is equivalent to coin flipping. A loss of 0.692 $\approx \ln 2$ means "learned nothing.")
How to read this:
- The plain 64-layer network failed to train. In this experiment, thanks to He initialization, the gradient did not become extremely small (the first-layer gradient was about 1/3 of that of the 4-layer network), yet the loss did not decrease. This is an optimization difficulty not explained solely by the "vanishing gradient" problem—in other words, the paradox from Section 1.
- Even with BN added, the plain 64-layer network failed, and its first-layer gradient exploded by a factor of 10,000. The fact that BN actually amplifies gradients in deep plain networks aligns with phenomena pointed out by subsequent theoretical studies (though those studies were not separately verified in this article).
- Residual + BN achieved an accuracy of 1.000 across all depths (4, 16, and 64) and its initial gradient was similar regardless of depth. The shortcut decoupled the magnitude of the gradient from the depth.
- Residual without normalization became unstable starting from 16 layers. Looking at the learning curves, there were cases where the loss decreased before suddenly spiking and stalling around 0.69 (seed 0, learning rate 0.03). This is likely because values added via shortcuts accumulate across layers, causing activations to grow. This result demonstrates why actual ResNets are used alongside BN.
Being a toy experiment, this cannot be generalized to actual ImageNet scale. However, the division of roles among "depth + shortcuts + normalization" points in the same direction as the paper.
5. Paper Figures
| Metric | Value (Paper) | Condition |
|---|---|---|
| ILSVRC 2015 Classification 1st Place | Top-5 Error 3.57% | ImageNet test server, ensemble of 6 models with varying depths |
| ResNet-152 single model | Top-5 4.49% | Validation set |
| Computational Cost | ResNet-152 11.3 billion FLOPs, VGG-19 19.6 billion | Based on multiply-add operations |
| CIFAR-10 | 110 layers 6.43% (best of 5 runs, average 6.61), 1202 layers 7.93% | 1202 layers had a training error of less than 0.1% — Overfitting |
| COCO Detection | mAP@[.5,.95] 21.2 $\rightarrow$ 27.2 (28% relative) | Faster R-CNN backbone changed from VGG-16 to ResNet-101, COCO validation set |
The reason ResNet-152 requires less computation than VGG-19 (19 layers) is due to bottleneck blocks . Channels are reduced using a 1$\times$1 convolution, processed with 3$\times$3, and expanded back with 1$\times$1. The expensive 3$\times$3 convolution operates only on a reduced number of channels. ResNet-50/101/152 use this block.
The final row has implications beyond the scope of this article. Simply plugging a classification-trained backbone into a detector dramatically improved detection performance. Since then, almost all detection and segmentation models (Parts 5, 6, and 8) start from ResNet-family backbones.
6. Limitations
- More depth does not endlessly improve performance. The 1202-layer model on CIFAR-10 performed worse than the 110-layer model. While it trained successfully (training error under 0.1%), the network was simply too large for the small dataset.
- BN relies on batches. Because it uses batch means and variances, small batch sizes cause statistics to fluctuate. This becomes problematic in detection and segmentation tasks where large image sizes limit GPUs to only a few images per batch, which is why alternatives like Group Normalization and Layer Normalization are used. Transformers in Part 7 use Layer Normalization.
- Inference and training behaviors differ. Batch statistics are used during training, while moving averages are used during inference. Forgetting to switch modes results in different answers for the same image (a common mistake in practice).
7. Connection to Our Project
- Shortcuts are now ubiquitous. The HRNet-family backbone for TT Score ball finding builds on stacked residual blocks, and the Transformer blocks used by SAM in Parts 9–11 are also residual connections of the form "$x + \text{Attention}(x)$" and "$x + \text{MLP}(x)$."
- This is meaningful even for small models. As shown in the experiment in Section 4, shortcuts stabilize training regardless of depth, allowing even a ball-finding model with around 1.5 million parameters to stack sufficient layers.
The next post covers structures that preserve resolution: U-Net and HRNet. We will examine why pooling in Part 1 becomes problematic when pinpointing tiny objects like a ball at the pixel level, and how to avoid it.
TT's Word. Instead of telling the deep network to "redraw everything," telling it to "tell me what to fix" allowed it to learn even 64 layers. That is the power of a single addition.
References
- He, K., Zhang, X., Ren, S., Sun, J. (2016). Deep Residual Learning for Image Recognition. CVPR 2016. https://arxiv.org/abs/1512.03385
- He, K., Zhang, X., Ren, S., Sun, J. (2016). Identity Mappings in Deep Residual Networks. ECCV 2016. https://arxiv.org/abs/1603.05027
- Ioffe, S., Szegedy, C. (2015). Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. ICML 2015. https://arxiv.org/abs/1502.03167
- Santurkar, S., Tsipras, D., Ilyas, A., Madry, A. (2018). How Does Batch Normalization Help Optimization? NeurIPS 2018. https://arxiv.org/abs/1805.11604
- Example code and output: in this folder,
code/02_residual.py,code/02_residual.out.txt