DETR: Detection as Set Prediction with Hungarian Matching and Anchor-Free Detection

Status: Draft (2026-10-13), partially measured. The Hungarian and GIoU examples are values run directly on a notebook CPU (code/08_hungarian.py, under 1 second). Paper figures were taken from the original text along with their conditions, and hardware is noted by tier. Writing rules: Measured data is the baseline, and unmeasured items are marked as "Unmeasured".

Table of Contents

• 1. Hand-Crafted Components
• 2. Architecture: N Query Slots
• 3. Training: 1:1 Matching
  • GIoU: Telling Distance Even Without Overlap
• 4. Hands-On: The Hungarian Algorithm
  • Why Greedy Search Fails
  • Implementation and Verification
• 5. Paper Metrics
• 6. Follow-Up Research: Mitigating Weaknesses
• 7. Limitations and Our Project
• References

1. Hand-Crafted Components

Detectors in parts 5 and 6 relied on many human-designed components: anchor sizes and ratios, IoU thresholds for assigning positive/negative labels to anchors, and NMS with its threshold for removing duplicates. While these components work when tuned properly, they must be readjusted when the data changes, and poor tuning quietly introduces blind spots ("0 positives" and "NMS removing player B" from part 5).

DETR (DEtection TRansformer) by Carion et al. changes the question. The ground truth of detection is a set of boxes. It has no order and no duplicates. If so, why not train the model to output a set right from the start, eliminating the need for both anchors and NMS?

2. Architecture: N Query Slots

1. CNN backbone(such as ResNet-50) generates a feature map.
2. Transformer encodermixes relationships between positions in the feature map using self-attention (Part 7).
3. Transformer decoderreceives learned N=100 query slots. Each slot views the encoder output via cross-attention and each other via self-attention.
4. A small MLP per slot outputs a class (including "no object") and a single box.

The fact that slots attend to one another is crucial. Since the network can learn that "another slot is handling that object, so I will handle something else," the NMS step for removing duplicates becomes unnecessary. In fact, the paper reported that applying NMS to the final layer output actually slightly decreases AP.

3. Training: 1:1 Matching

There are 100 outputs and usually a few ground-truth items. Which outputs should be compared against which ground-truth items to calculate the loss? DETR finds the 1:1 match with the lowest cost.

짝 비용(정답 i, 슬롯 j) = −p_j(정답 i 의 클래스) + 5 · L1(상자) + 2 · (1 − GIoU(상자))

(Weights λ_L1 = 5, λ_iou = 2 are paper values.) Once matches are determined, matched slots learn the target class and box, while unmatched slots learn "no object." Because "no object" applies to the vast majority of slots, its loss is scaled down by a factor of 10 (addressing the imbalance issue similar to focal loss in Part 4). Auxiliary losses are also applied to all decoder layer outputs.

This matching is the classic assignment problem, solved using Kuhn's Hungarian algorithmfrom 1955.

GIoU: Telling Distance Even Without Overlap

IoU is 0 regardless of how far apart two non-overlapping boxes are. Using it as a loss yields no gradients. GIoU(Generalized IoU) penalizes the "empty space inside C" by utilizing the smallest enclosing box C that bounds both boxes.

GIoU = IoU − |C 에서 두 상자의 합집합을 뺀 넓이| / |C|      (−1 ~ 1)

Manual calculation yields the following (measured):

While IoU is 0 for both distant cases, GIoU becomes smaller as the distance increases, creating a gradient that pulls the box toward the ground truth.

4. Hands-On: The Hungarian Algorithm

Why Greedy Search Fails

Here are the costs for three slots in a scene where two players overlap. Slot 0 straddles both players and fits both moderately well, Slot 1 fits player A reasonably well, and Slot 2 fits player B quite well.

          슬롯 0   슬롯 1   슬롯 2
선수 A     0.9      1.0      6.0
선수 B     0.8      6.0      3.0

If we greedily pick "the cheapest remaining slot for each ground truth in turn," A takes Slot 0 (0.9) first, and B receives Slot 2 (3.0), resulting in a total cost of 3.9 . The Hungarian algorithm finds A–Slot 1 and B–Slot 0 for a total cost of 1.8 (measured). The strategy where whoever picks first wins can be detrimental overall. Using greedy matching for training would cause Slot 2 to mistakenly learn player B and Slot 1 to learn "no object."

Implementation and Verification

We implemented an O(n²m) version in numpy using potentials (dual variables) (ground truth n ≤ slots m). The core idea is to "increase matches row by row, but if a cheaper path (augmenting path) exists that alters existing matches, flip the matches along that path."

Evaluating all permutations for just 10 ground truths × 100 slots yields 6.3×10¹⁹ possible combinations. Hungarian-family algorithms solve this in polynomial time (roughly n³). A single DETR frame (dozens of ground truths × 100 slots) takes milliseconds, meaning it is not a training bottleneck. In practice, SciPy's linear_sum_assignment(modified Jonker–Volgenant algorithm) is frequently used.

Matching 3 ground truths (two players, ball) and 6 slots using the DETR cost formula (class probability + 5·L1 + 2·(1−GIoU)) assigns each ground truth to the most plausible slot, while the remaining 3 slots are assigned as "no object" (code/08_hungarian.out.txt).

5. Paper Metrics

(Paper Table 1. Faster R-CNN version uses GIoU loss, random crop augmentation, and an enhanced version trained for ~109 epochs. Speed measured with a TorchScript model, GPU not specified.)

• Large objects improve by +7.8, while small objects drop by −5.5. The encoder's global attention views the entire image making it strong on large objects, but because the feature map has only a single resolution (1/32), it is weak on small objects. It lacks multi-scale feature representations like FPN in Part 5.
• Training takes a long time. Comparison models take 500 epochs, ablation study models take 300 epochs, and 300 epochs took 3 days on 16 datacenter GPUs. Early in training, the pairing between slots and ground truths changes at every step so slots haven't settled on what to specialize in, resulting in slow convergence.

6. Follow-Up Research: Mitigating Weaknesses

DETR's "set prediction + 1:1 matching" became the standard paradigm for subsequent detection. Both YOLOv10's NMS-free method from Part 6 (one-to-one head) and SAM 3's detector from Part 11 (DETR-family) build upon this idea.

7. Limitations and Our Project

• Small objectsremain difficult. Although subsequent models have drastically mitigated this, finding table tennis balls that are only a few pixels wide using set prediction is less advantageous than heatmap approaches (Part 3).
• Matching dictates training. Cost weights (class 1, L1 5, GIoU 2) determine "what counts as the same object." Changing these values alters what the model learns.
• In our project, we do not yet directly use DETR-family models. However, SAM 3's detector, which was a candidate for locating table tennis tables via text prompts, belongs to the DETR family (Part 11), and RT-DETR (Apache-2.0)—which can be deployed without NMS when human/racket bounding boxes are needed in the future—is a candidate to avoid the AGPL issues of Part 6 (unmeasured).

Starting from the next post, we cover segmentation: Segment Anything (SAM), which paints anything with a single click.

Titi's Note. Picking the cheapest sequentially yields 3.9, while Hungarian yields 1.8. A strategy where whoever picks first takes everything is detrimental in team sports.

References

• Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., Zagoruyko, S. (2020). End-to-End Object Detection with Transformers. ECCV 2020. https://arxiv.org/abs/2005.12872
• Kuhn, H. W. (1955). The Hungarian method for the assignment problem. Naval Research Logistics Quarterly, 2(1–2), 83–97. https://doi.org/10.1002/nav.3800020109
• Rezatofighi, H. et al. (2019). Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regression. CVPR 2019. https://arxiv.org/abs/1902.09630
• Zhu, X. et al. (2021). Deformable DETR: Deformable Transformers for End-to-End Object Detection. ICLR 2021. https://arxiv.org/abs/2010.04159
• Li, F. et al. (2022). DN-DETR: Accelerate DETR Training by Introducing Query DeNoising. CVPR 2022. https://arxiv.org/abs/2203.01305
• Zhang, H. et al. (2023). DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection. ICLR 2023. https://arxiv.org/abs/2203.03605
• Zhao, Y. et al. (2024). DETRs Beat YOLOs on Real-time Object Detection. CVPR 2024. https://arxiv.org/abs/2304.08069
• SciPy linear_sum_assignment: https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.linear_sum_assignment.html
• Summary of Time Complexity for the Hungarian Algorithm: https://en.wikipedia.org/wiki/Hungarian_algorithm
• Example Code and Output: In this folder, code/08_hungarian.py, code/08_hungarian.out.txt