Object Detection 1: From Two-Stage Detectors R-CNN to Faster R-CNN, and Anchors

Status: Draft (2026-10-13), partially measured. IoU, anchor, and NMS examples are measured directly on a laptop CPU (code/05_boxes.py, under 1 second). Paper figures are taken from the original text along with their conditions. Hardware is noted by tier. Writing rule: Measured data is the default; unmeasured items are marked as "unmeasured".

Table of Contents

• 1. What is Detection, and IoU
• 2. R-CNN (2014): CNN per Proposal
• 3. Fast R-CNN (2015): CNN Once for the Whole Image
• 4. Faster R-CNN (2015): The Network Also Finds Proposals
  • Anchors
  • Try It Yourself: Laying Anchors and Assigning Labels
  • NMS: Removing Overlapping Proposals
  • Results (Paper)
• 5. Next: FPN
• 6. Limitations
• 7. Connection to Our Project
• References

1. What is Detection, and IoU

Classificationasks "what is in this image," while detectionasks "what is where and how many." The outputs for each object are a bounding box (x1, y1, x2, y2) , a class, and a confidence score.

How well a box matches is measured by IoU(Intersection over Union). It is the area of overlap between two boxes divided by their union area, where 1 means complete overlap and 0 means no overlap. Calculated by hand, it looks like this (measured).

Even with a "half overlap," the IoU is not 0.5 but 0.33. Just a 1-pixel shift drops it to 0.68. The fact that smaller objects make IoU much more sensitive to a few pixels of error is one reason why bounding box detection is difficult for tiny objects like a ping-pong ball. The "AP" and "mAP" in detection papers are the area under the precision-recall curve calculated by treating predictions that exceed an IoU threshold (typically 0.5, or the 0.5–0.95 average for COCO) as correct (we will calculate this ourselves in Part 6).

The fundamental difficulty of detection is that the number and size of objects are unknown in advance. Unlike classification, the number of output slots cannot be fixed. Two-stage detectors answer this by saying: "First, generously extract proposals where objects are likely to be, and run a classifier on each proposal."

2. R-CNN (2014): CNN per Proposal

R-CNN by Girshick et al. is the first success in bringing the CNN features demonstrated by AlexNet in Part 1 to detection.

1. Proposal Boxes: About 2,000 proposals per image are extracted using a classical algorithm (selective search) that groups regions with similar colors and textures.
2. CNN Features: Each proposal image is cropped and resized to 227×227 (including a 16px surrounding context padding) and fed into the CNN.
3. Classification and Refinement: Linear SVMs determine the class, and a separate regression is trained to slightly adjust the box coordinates. Box refinement alone raised mAP by 3–4 points.

Results (Paper): PASCAL VOC 2010 mAP 53.7%, VOC 2012 53.3% (over 30% relative improvement over previous bests), ILSVRC2013 detection 31.4% (OverFeat 24.3%). However, it took 13 seconds per image on a GPU and 53 seconds on a CPU. This is because the CNN is run separately for each of the 2,000 proposals. It is essentially recalculating the same pixels 2,000 times for overlapping proposals.

3. Fast R-CNN (2015): CNN Once for the Whole Image

The solution is to change the order. Pass the entireimage through the CNN once to create a feature map, and crop proposal boxes from that feature map .

• RoI Pooling: Divides variably-sized proposal regions on the feature map into a fixed size (7×7 for VGG16) and extracts the maximum value for each cell. This is required to feed them into subsequent fully connected layers.
• Multi-task Loss: Combines classification loss and box regression loss (smooth L1: squared when error is less than 1, absolute value otherwise) to train together in a single network. It unified the multiple stages of R-CNN, which trained SVMs and regression separately.

Results (Paper, VGG16, single datacenter GPU): 9.5 hours for training (84 hours for R-CNN), 0.32 seconds per test image ( 213x faster than R-CNN). VOC 2007 mAP 66.9% (07 data) · 70.0% (07+12), VOC 2012 65.7% · 68.4%. However, the remaining bottleneck was no longer the CNN, but selective search for extracting proposal boxes . In the table of the next paper (Faster R-CNN), selective search + Fast R-CNN takes 1,830ms per image, while the Fast R-CNN core takes just around 0.3 seconds, meaning almost all the rest goes to finding proposals.

4. Faster R-CNN (2015): The Network Also Finds Proposals

Ren et al. replaced proposal extraction itself with a small network on top of the CNN features, called the RPN(Region Proposal Network). Since the RPN shares the same feature map with the detector, proposals are obtained almost for free.

Anchors

: The RPN uses predefined boxes called anchors at each cell of the feature map Reference boxes (anchors) are placed in quantities of k. The default is sizes of 128²·256²·512² × aspect ratios of 1:1·1:2·2:1 = 9 boxes. The model outputs two things for each anchor:

1. Whether an object exists in this anchor (object/background score)
2. How much to shift and scale the anchor to fit the object (4 correction values)

The key idea is to predict "how much to adjust from the reference box" instead of guessing the box from scratch. Since large, small, and long objects each start from similar anchors, learning is much easier.

Try it yourself: Laying anchors and labeling

We laid anchors on a 1000×600 image with a feature map stride of 16 (measured).

The last row seems to vary depending on detailed rules that determine anchor center and size (rounding, where the center is placed within a cell), and the exact cause was not separately verified.

The labeling rule is that an IoU above 0.7 is positive (object), below 0.3 is negative (background), and anything in between is excluded from training. We placed a single object of width 180 × height 240 (e.g., the upper body of an athlete on screen) and tried labeling it.

IoU>0.7 양성 0개, IoU<0.3 음성 7,502개, 사이(무시) 542개, 가장 잘 맞는 앵커 IoU 0.659
가장 잘 맞는 앵커 [232, 168, 488, 424] → 보정값 tx,ty,tw,th = [0.117, 0.016, -0.352, -0.065]

There are 0 positives. If the size and aspect ratio of the 9 types of anchors do not match this object well, there might not be any anchor exceeding 0.7 IoU. Therefore, the paper includes a rule stating, "For each ground truth box, the anchor with the highest IoU is also counted as positive." Without this rule, this object would be entirely omitted from training. This is an example showing that anchor design (size and aspect ratio) must match the shape of objects in the data, and that gaps when they don't match are filled in by rules.

The correction values are in the form of tx = (정답 중심 x − 앵커 중심 x) / 앵커 너비, tw = log(정답 너비 / 앵커 너비) . By dividing by the anchor size and applying a logarithm, the errors for large and small objects become numbers of similar magnitudes.

NMS: Removing overlapping candidates

Since multiple similar candidates appear for a single object, they are sorted out using Non-Maximum Suppression(NMS). The box with the highest score is kept, any remaining boxes with an IoU exceeding the threshold are deleted, and the process is repeated. We tested this with four candidates from a scene where two athletes partially overlap (0 and 1 for athlete A, 2 and 3 for athlete B) (measured).

A single threshold cannot simultaneously separate "different overlapping objects" and "duplicates of the same object." This is why NMS is weak in crowded scenes, and it is also the motivation behind DETR in part 8 trying to eliminate NMS.

Results (Paper)

RPN reduces candidates to about 2,000 using NMS (threshold 0.7), and uses only 300 during testing. It was more accurate with fewer candidates than selective search's 2,000.

5. Next: FPN

If object sizes vary, a single feature map is not enough. FPN (Feature Pyramid Network, 2017) scales up deep-layer features by 2x while coming down (top-down), matches shallow-layer features of the same resolution using 1×1 convolutions, and creates a summing pyramid. Small objects are found in high-resolution layers, and large objects are found in low-resolution layers. The concept is similar to U-Net in part 3. Attaching FPN (ResNet-101) to Faster R-CNN yielded a COCO test-dev AP of 36.2 (single model).

6. Limitations

• Slow. The two-stage approach (candidates → classification) is accurate but too heavy for real-time use. 5fps falls far short of table tennis footage at 60 frames per second. One-stage detectors in part 6 target this problem.
• Many hand-crafted settings. Anchor sizes and ratios, IoU 0.7/0.3 thresholds, and NMS thresholds are all manually determined by humans. Just like the experiment in section 4, if these values do not match the data, blind spots quietly emerge.
• Small objects. A single cell in a feature map with a stride of 16 is 16×16 pixels. A table tennis ball with a diameter of 3–12px is smaller than a single cell.

7. Connection to our project

TT Score finds the ball using a heatmaprather than a box(Part 3), because all three of the above limitations apply to table tennis balls. Instead, the box detector is used to find people. Precise pose analysis uses a \u201ctop-down\u201d approach that first finds a bounding box for a person using a human detector (RTMDet) and then crops that box to find joints (TT Score Model Anatomy Part 4). The structure of detector \u2192 crop for precise determination is the same concept as a two-stage detector.

TT Note. Even with a 50% overlap, the IoU is 0.33. Box matching is a stricter test than you might think.

References

• Girshick, R., Donahue, J., Darrell, T., Malik, J. (2014). Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation. CVPR 2014. https://arxiv.org/abs/1311.2524
• Girshick, R. (2015). Fast R-CNN. ICCV 2015. https://arxiv.org/abs/1504.08083
• Ren, S., He, K., Girshick, R., Sun, J. (2015). Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. NeurIPS 2015. https://arxiv.org/abs/1506.01497
• Lin, T.-Y. et al. (2017). Feature Pyramid Networks for Object Detection. https://arxiv.org/abs/1612.03144
• Example code and output: In this folder, code/05_boxes.py, code/05_boxes.out.txt