What Convolutional Neural Networks Learn: Filters, Receptive Fields, Backpropagation, and LeNet & AlexNet

Status: Draft (2026-10-13), partially measured. The examples were run directly on a laptop CPU (Python 3.9, numpy 2.0). The code is in this folder's code/01_conv.py and finishes in a few seconds. Paper metrics were transcribed from the original texts along with their conditions. Writing rules: Measured data is the default, and unmeasured values are marked as "Unmeasured".

Table of Contents

• 1. Why Images Need a Special Structure
• 2. Convolution: Sliding a Small Window to Multiply and Add
• 3. Filters Are Not Hand-Crafted: Loss, Backpropagation, Gradient Descent
  • Try It Yourself: From a Random Filter to a Sobel Filter
• 4. Pooling and Receptive Fields: The Scope of a Single Unit
• 5. LeNet-5 (1998): The Prototype of Convolution + Pooling + Backpropagation
• 6. AlexNet (2012): Same Idea, Different Scale
• 7. Limitations
• 8. Connection to Our Project
• References

1. Why Images Need a Special Structure

A single mobile phone photo contains millions of numbers (pixels). Feeding this into a regular neural network, namely a fully connected layer(where every input connects to every output), causes the number of connections to explode. Consider just a single layer that transforms a 32×32 grayscale image into six 28×28 feature maps:

(Direct calculation, code/01_conv.py [5]. Convolution is 6 filters × (25 + 1 bias).)

The 30,000-fold difference is due to two properties inherent in images:

1. Pixels close to each other are deeply related (Locality). Pixels on the opposite corner are unnecessary for recognizing the outline of a ball.
2. The same pattern is the same pattern no matter where it appears (Translation invariance). A ball on the left side of the screen and one on the right can be found using the same method.

Convolution is an operation built by baking these two assumptions into its structure. It looks only through a small window (locality) and reuses the same filter across the entire image (weight sharing). This practice of "building assumptions into the structure" is called inductive bias, and in Part 7 we will see what price the Vision Transformer pays by discarding this.

2. Convolution: Sliding a Small Window to Multiply and Add

The calculation is simple. Overlapping a 3×3 filter (9 numbers) with a 3×3 window of the image, multiplying corresponding positions, and summing them all up yields a single output unit. Repeating this while shifting the window one step at a time completes the output map.

# code/01_conv.py 에서
def conv2d(x, k):
    kh, kw = k.shape
    out = np.empty((x.shape[0] - kh + 1, x.shape[1] - kw + 1))
    for i in range(out.shape[0]):
        for j in range(out.shape[1]):
            out[i, j] = np.sum(x[i:i + kh, j:j + kw] * k)
    return out

Applying a vertical edge filter (Sobel filter) to a 6×6 image where the left half is 0 (dark) and the right half is 1 (bright) yields the following (measured):

필터            출력(4×4)
[-1  0  1]      [0 4 4 0]
[-2  0  2]      [0 4 4 0]
[-1  0  1]      [0 4 4 0]
                [0 4 4 0]

4 appears only at the boundary where brightness changes, and 0 elsewhere. A filter is a question asking, "Does this pattern exist here?", and the output map spreads the answer to that question across the entire image. Real networks place dozens to hundreds of such filters in a single layer, with each filter searching for a different pattern (horizontal edges, diagonals, color blobs, etc.).

Additionally, what deep learning frameworks call "convolution" is actually cross-correlation, which does not flip the filter unlike the convolution found in math textbooks. Because filters are learned anyway, whether they are flipped makes no difference.

3. Filters Are Not Hand-Crafted: Loss, Backpropagation, Gradient Descent

In traditional computer vision, humans designed filters like the Sobel filter above. The core of CNNs is that filter numbers are learned from data. The learning process has three steps:

1. Loss function: Measures how much the output produced by the current filter differs from the target answer using a single number. Here, we use Mean Squared Error (MSE).
2. Backpropagation: Calculates how much the loss changes (the gradient) if each of the 9 filter numbers is slightly altered. In convolution, convolving the "output error" and the "input" yields the filter gradients.
3. Gradient descent: Slightly shifts the filter in the opposite direction of the gradient. k ← k − 학습률 × 기울기.

# code/01_conv.py 에서 — 필터 k 의 기울기
for x, y in zip(X, Y):
    err = conv2d(x, k) - y                       # 출력 오차
    for a in range(3):
        for b in range(3):
            g[a, b] += np.mean(2 * err * x[a:a + err.shape[0], b:b + err.shape[1]])
k -= lr * g

Try It Yourself: From a Random Filter to a Sobel Filter

We passed 32 random images through a Sobel filter to create "target outputs" and performed gradient descent starting from a random filter (mean 0, standard deviation 0.1) with a learning rate of 0.1 without revealing the target filter (measured, laptop CPU approx. 6 seconds).

The filter learned after 500 steps was [[-1, 0, 1], [-2, 0, 2], [-1, 0, 1]] , and the maximum difference from the target was 0.0005. Even though no human instructed it to "find vertical edges," the data demanded such a filter, causing it to take that shape. This is why plotting the first-layer filters of a real CNN reveals edge and color pattern shapes.

To verify that the gradients computed via backpropagation are correct, we compared them against numerical differentiation(a method of directly measuring loss changes by tweaking a single number ever so slightly). They matched with a maximum relative difference of 6.4×10⁻¹¹. This is always a worthwhile check before trusting self-written training code.

4. Pooling and Receptive Fields: The Scope of a Single Unit

A single output unit of a convolutional layer looks at only a 3×3 patch of the input. To recognize an entire ball or an athlete's arm, we need a wider view. There are two ways to achieve this:

• Stacking layers. Placing a 3×3 layer on top of another 3×3 layer means a unit in the upper layer sees a 3×3 patch of the lower layer, which corresponds to a 5×5 patch of the input.
• Downsampling (Pooling/Stride). Combining a 2×2 block into a single unit (via max or average) allows the same filter to cover a twice-as-wide range.

The range of the input that affects a single output unit is called the receptive field. It is calculated per layer as r ← r + (커널 − 1) × j, j ← j × 보폭 (wherej is the cumulative stride). Calculated using the architectures of LeNet-5 and AlexNet (direct calculation, code/01_conv.py [4]).

Pooling comes at a cost. Reducing resolution causes the model to lose exact locations. While this poses no problem for classification ("Is there a cat in this image?"), it is fatal for tasks requiring pixel-level ball coordinates. Part 3 (U-Net & HRNet) tackles this exact problem.

5. LeNet-5 (1998): The Prototype of Convolution + Pooling + Backpropagation

LeCun et al.'s 1998 paper introduced LeNet-5 for reading handwritten digits (MNIST). The input is a 32×32 grayscale image. Although the digits in the data are at most 20×20 and centered within a 28×28 grid, the input was intentionally made larger than that (which aligns with the receptive field calculation where the final layer views the entire 32×32 input).

• Architecture: Convolution (C1) → Subsampling (S2) → Convolution (C3) → Subsampling (S4) → Convolution (C5) → Fully Connected → Output. That is 7 layers excluding the input.
• Scale: There are 340,908 connections, but only about 60,000 numbers are learned, thanks to weight sharing.
• Result (Paper): MNIST test set error rate 0.95%. When 540,000 images generated via geometric deformations (translation, scaling, distortion, skewing) were added to the 60,000 training images, it dropped to 0.8% .

This paper demonstrated that features do not need to be hand-crafted, but can be learned end-to-end using convolutional structures and backpropagation. However, the computational power and data available in 1998 struggled to scale beyond small, clean images like handwritten digits.

6. AlexNet (2012): Same Idea, Different Scale

Fourteen years later, AlexNet by Krizhevsky, Sutskever, and Hinton won the ImageNet competition (ILSVRC, 1000 classes, approx. 1.2 million training images) by a wide margin. The architectural idea is the same as LeNet's. What changed were the scale and several training techniques.

"Top-5 error" is the rate at which the correct answer is not among the top 5 choices selected by the model. The fact that 15.3% is an ensemble metric is a condition frequently omitted.

Looking at what the authors directly measured to see what made a difference reveals "why it was designed this way."

• ReLU. Replacing the activation function from tanh to max(0, x) made a 4-layer CNN reach a training error of 25% on CIFAR-10 6 times faster (Paper Figure 1). While tanh gradients approach 0 for large values, ReLU maintains a constant positive gradient of 1. This difference was decisive for training large networks within a limited time.
• Data Augmentation. Randomly cropping 224×224 patches from 256×256 images and horizontally flipping them virtually increased the training data by a factor of 2048. Color augmentation via principal component analysis (PCA) color shifting reduced the top-1 error by over 1%. The paper explains that without augmentation, 60 million parameters would simply memorize the data.
• Dropout. Shuts off fully connected neurons with a probability of 0.5 during training. This prevents neurons from co-adapting, thereby reducing overfitting. However, it roughly doubles the number of iterations required for convergence.
• Overlapping pooling(window 3, stride 2) reduced top-1/top-5 errors by 0.4%/0.3%, and local response normalizationreduced them by 1.4%/1.2%. Local response normalization was later superseded by batch normalization (Part 2) and is rarely used today.
• GPU. Because convolution repeats the same calculations countless times, it fits GPU parallel processing well. Splitting the network across two GPUs due to memory constraints is also a famous highlight of this paper.

In summary, the lesson of AlexNet is not a "single new idea," but a combination of large data + fast computation + techniques that make large networks trainable. Most vision models of the subsequent decade are built upon this formula.

7. Limitations

• Loss of location. Repeated pooling is good for classification, but blurs precise localization (Part 3).
• Hard to stack deeply. AlexNet had 8 layers. Stacking deeper led to training failure, a problem solved by ResNet (Part 2).
• Data-hungry (requires many labels). Just like ImageNet, hundreds of thousands to millions of human-labeled images were required. The path of learning without labels is covered in a sister series on self-supervised learning.
• Vulnerable to scale and rotation. While convolutions are robust to translation, they do not structurally guarantee handling objects that become larger or rotate. Therefore, we teach the model through augmentation.

8. Connection to Our Project

The ball detection models of TT Score (WASB and BlurBall) will still be convolutional neural networksin 2026. The reason is precisely what was covered in this post.

• The ball is small (roughly 3px in diameter at the model input) and blurs quickly. The inductive bias of a "small window + weight sharing" is well-suited for finding such patterns.
• A small network with about 1.5 million parameters is sufficient, processing a single window in 10.5ms (measured on a server GPU). For real-time judgment, this speed is just as important as accuracy.

Detailed architectures and measurements can be found in TT Score Model Anatomy Part 2: Ball Detection. The next post covers residual connections and batch normalization, which made deep networks trainable.

TT's Word. We never told the filters "find vertical edges," yet after 500 steps, they turned into Sobel filters. The filters automatically transform into the shapes desired by the data.

References

• LeCun, Y., Bottou, L., Bengio, Y., Haffner, P. (1998). Gradient-Based Learning Applied to Document Recognition. Proceedings of the IEEE, 86(11), 2278–2324. https://doi.org/10.1109/5.726791
• Krizhevsky, A., Sutskever, I., Hinton, G. E. (2012). ImageNet Classification with Deep Convolutional Neural Networks. NeurIPS 2012. https://papers.nips.cc/paper_files/paper/2012/hash/c399862d3b9d6b76c8436e924a68c45b-Abstract.html
• Example code and outputs: code/01_conv.py, code/01_conv.out.txt