Open-Vocabulary Detection: Attaching CLIP Text Embeddings to Detectors — OWL-ViT, OWLv2, Grounding DINO, YOLO-World, YOLOE

Status: Draft (2026-10-13), partially measured. The classification head example merely mimics the structure without real CLIP weights (code/12_open_vocab_head.py, under 1 second on a laptop CPU). The models in this post were not run in our environment (unmeasured). Paper metrics are transcribed from the original texts along with their conditions, and hardware is denoted by tier. The learning principles of CLIP itself are covered in the sister series (self-supervised, video, pose, sports). Writing rules: Measured data is the default, and unmeasured items are marked as "Unmeasured".

Table of Contents

• 1. The Wall of Closed Vocabulary
• 2. The Key: CLIP Text Embeddings
  • Try It Yourself: Adding a Class = Adding a Row
• 3. OWL-ViT (2022): One Box per ViT Token
• 4. OWLv2 (2023): Scaling Up with Self-Labeling
• 5. GLIP & Grounding DINO: Deeply Fusing Text and Images
• 6. YOLO-World & YOLOE: Going Real-Time
• 7. Summary: How to Choose
• 8. Limitations and Our Project
• References

1. The Wall of Closed Vocabulary

The detectors in parts 5, 6, and 8 are closed-vocabularymodels. When trained on COCO, they only find 80 categories (person, car, tennis racket...). To detect a "ping-pong racket" or "ping-pong table," you must collect data labeled with those terms and retrain.

From a structural standpoint, the reason is simple. The final classification head of a detector has the form (영역 특징) · Wᵀ , where W is a fixed matrix(80×d) with one row per class. Adding a new class requires an additional row in W, the values of which can only be obtained through data training.

2. The Key: CLIP Text Embeddings

CLIP (2021) is a model trained on 400 million internet image-caption pairs such that its image encoder and text encoder output vectors in the same space. It uses contrastive learningto bring matching images and texts close together while pushing non-matching ones apart. As a result, the text vector for "a photo of a racket" and the image vector of a racket photo become close. Using this text vector as the classifier's weights achieved 76.2% ImageNet top-1 zero-shot without using a single ImageNet training image (ViT-L/14, 336px). This is on par with ResNet-50 trained directly on ImageNet.

Why CLIP learns this way is covered in the sister series; this post only looks at how these text vectors are attached to the detector. The idea boils down to a single line: Instead of learning the rows of W, generate them by passing class names into the text encoder.

닫힌 어휘:  점수 = r · W[클래스]ᵀ               (W 는 학습한 80×d 행렬)
열린 어휘:  점수 = r · 글자인코더("racket")ᵀ / τ  (어휘는 쓸 때 적는다)

Try It Yourself: Adding a Class = Adding a Row

Instead of real CLIP, a random 2-layer MLP was used to mimic the text encoder, and a classification head was attached to an 80×80 feature map (640px input, corresponding to a stride of 8) (measured; the numbers themselves have no meaning, we only observe the structure).

어휘 3개 → 분류 머리 행렬 (3, 256)
「racket」을 더하면 (4, 256) — 다른 가중치는 그대로, 출력 칸마다 로짓이 4개
두 방식 출력 최대 차 0.0e+00, 프레임당 9.09 ms → 2.71 ms

The last row compares the approach of feeding the vocabulary into the text encoder on every frame versus "pre-cooking" the vocabulary once into a matrix. The answers are entirely identical, and only time is reduced. Since the mock encoder is very small, the difference is only about 3x, but with a real CLIP text encoder being a Transformer, this difference will likely be larger (unmeasured). This is what YOLO-World and YOLOE in Section 6 aim for.

3. OWL-ViT (2022): One Box per ViT Token

OWL-ViT by Minderer et al. demonstrated the simplest method.

1. 1. Take a contrastively trained ViT (Part 7) like CLIP.
2. 2. Remove the final pooling layer that aggregates tokens into one. As a result, outputs remain for each patch token.
3. 3. Attach a lightweight box head(bounding box of the object assigned to this token) and a classification head(token embedding) to each token. The weights of the classification head are the text embeddings of the class names.
4. 4. Fine-tune on detection data (a mix of two out of OpenImages, Objects365, and Visual Genome).

Much like DETR in Part 8, a single token (slot) is responsible for a single object, and matching also uses the DETR-style set loss. The difference is that instead of having separate slots, image patch tokens serve directly as slots.

Result (paper): AP of 34.6 on LVIS v1.0 validation, with rare class AP of 31.2(ViT-L/14, trained on Objects365+VG, 840px). Since rare class names were omitted from training, 31.2 represents the zero-shot value for previously unseen classes.

You can also query using example imagesinstead of text. Using the embedding of an image patch as a query translates to "find something similar to this." On unseen COCO classes, the one-shot detection AP50 rose from the previous best of 26.0 to 41.8.

4. OWLv2 (2023): Scaling Up with Self-Labeling

Detection labels are expensive. Minderer et al. overcome this wall in OWLv2 through self-training.

1. 1. Annotate the entire WebLI web image dataset (approx. 10 billion image-alt-text pairs) with pseudo box labelsusing the existing OWL-ViT (CLIP L/14). The vocabulary consists of terms extracted from the images' alt text.
2. 2. Train a new model (OWL-ST) using these pseudo labels. The training examples exceed 1 billion.
3. 3. Fine-tune on detection data.

Result (paper): LVIS rare class AP reached 31.2% → 44.6%(L/14, 43% relative improvement), while the larger G/14 reached 47.2%. As the scale grew, training efficiency was also tuned: combining an "objectness" head that computes boxes only for the top-k tokens, discarding 50% of patches with low pixel variance during training, and stitching multiple images into mosaics cut training computation roughly in half and doubled throughput.

This is the same concept as the data engine of SAM in Part 9: The model generates labels, and those labels build a better model. Part 11's SAM 3 compared its PCS accuracy against this OWLv2.

5. GLIP & Grounding DINO: Deeply Fusing Text and Images

OWL-ViT meets image features and text embeddings with only a final single dot product. This falls short when relationships within a sentence matter, such as "the racket held by the player on the left." Another branch fuses the two deeply from the start.

• GLIP (2022): Redefined detection as phrase grounding(finding regions matching phrases in a sentence). Pre-trained on 3 million human-annotated grounding data + 24 million web image-text pairs, GLIP-L achieved 49.8 AP on COCO validation without seeing COCO images, and 26.9 AP on LVIS.
• Grounding DINO (2024): Deeply integrated text into the DINO detector from Part 8. It features a feature enhancer (image self-attention + text→image and image→text cross-attention), language-guided query selection(initializing queries from image locations that align well with text), and a decoder that views both modalities together. The Swin-L version (trained on Objects365, OpenImages, and GoldG) achieved a COCO zero-shot 52.5 AP, and 63.0 AP on test-dev when fine-tuned. It also reported a zero-shot average of 26.1 AP on ODinW, a collection of multi-domain datasets; however, the model producing this score includes COCO in its training data. The code is under Apache-2.0.

The price of deep fusion is speed. Since image-side computations must be rerun whenever the text changes, vocabularies cannot be pre-cooked.

6. YOLO-World & YOLOE: Going Real-Time

This is an attempt to bring open vocabulary to the speed of the YOLO family from Part 6.

YOLO-World (2024): Employs RepVL-PAN, which fuses text embeddings into YOLO's feature pyramid, along with a region-text contrastive loss. The key is **"Prompt first, detect later."** Once the user defines the vocabulary, it is encoded once and re-parameterized into the network weights, allowing the text encoder to be completely stripped away during inference. This is the "pre-cooking" from the Section 2 example.

Result (paper Table 2): LVIS minival zero-shot 35.4 AP, 52.0 FPS. This figure comes with many conditions attached: the largest YOLO-World-L (48 million parameters after pre-cooking, 110 million before, 17.6 FPS), trained on Objects365 + GoldG + 246k pseudo-labeled CC3M images, measured on a single datacenter GPU without FP16 or TensorRT. The official repository is GPL-3.0 , and commercial use requires separate inquiries.

YOLOE (2025): Incorporates three types of prompts into a single model:

• RepRTA(Text): Merges a small auxiliary network refining text embeddings as parameters into the classification head for zero inference cost
• SAVPE(Example Image): A visual prompt encoder that separates meaning and location
• LRPC(Prompt-free): Self-selection from a large built-in vocabulary

Result (paper): Compared to YOLO-Worldv2-S, LVIS minival +3.5 AP, about 1/3 the training cost, and 1.4x faster inference (inference datacenter GPU, TensorRT). The official repository is AGPL-3.0 .

Both models are also included in the Ultralytics package (AGPL-3.0), inheriting the AGPL conditions from Section 5 of Part 6.

7. Summary: How to Choose

8. Limitations and Our Project

• Ambiguity of language. "table" is a dining table, and it needs to be "table tennis table" for a ping-pong table. "paddle" can also mean a canoe oar. With open vocabulary, choosing the vocabulary is prompt design itself.
• Small objects. These models generally treat the entire image as a patch/feature pyramid, so a ping-pong ball that is only a few pixels wide remains difficult. Finding the ball is handled by a dedicated heatmap model (Part 3).
• Conditions for zero-shot numbers. The criteria for "unseen classes" vary by paper (whether only the name was excluded from training, or if the dataset images were not seen at all). In some cases, such as 26.1 in ODinW, training data overlaps with the evaluation data.
• In TT Score, We do not yet use open-vocabulary detectors. Tables are found using human taps and SAM 2.1, and text-based finding is planned with SAM 3 as a candidate (Part 11). When only a "few fixed things" are needed, such as people and rackets, closed-vocabulary models are often faster and more accurate. Open-vocabulary shines when "the user decides what to look for," for instance, when a coach wants to find a "shuttlecock tube" or "ball basket" on the fly (unmeasured, idea stage).

This post concludes Part 1 of the series. Self-supervised learning without name tags, CLIP, and video, pose, and sports models continue in the sister series (Self-Supervised, Video, Pose, and Sports Folder).

TT in a Word. To teach an older detector what a "racket" is, you had to gather a bunch of racket photos. Now, just the single word "racket" is enough. However, if you just say "table", it might bring up a dining table, so you have to choose your words carefully.

References

• Radford, A. et al. (2021). Learning Transferable Visual Models From Natural Language Supervision. ICML 2021. https://arxiv.org/abs/2103.00020
• Minderer, M. et al. (2022). Simple Open-Vocabulary Object Detection with Vision Transformers. ECCV 2022. https://arxiv.org/abs/2205.06230
• Minderer, M., Gritsenko, A., Houlsby, N. (2023). Scaling Open-Vocabulary Object Detection. NeurIPS 2023. https://arxiv.org/abs/2306.09683
• Li, L. H. et al. (2022). Grounded Language-Image Pre-training. CVPR 2022. https://arxiv.org/abs/2112.03857
• Liu, S. et al. (2024). Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection. ECCV 2024. https://arxiv.org/abs/2303.05499
• Cheng, T. et al. (2024). YOLO-World: Real-Time Open-Vocabulary Object Detection. CVPR 2024. https://arxiv.org/abs/2401.17270, repository https://github.com/AILab-CVC/YOLO-World (GPL-3.0)
• Wang, A. et al. (2025). YOLOE: Real-Time Seeing Anything. ICCV 2025. https://arxiv.org/abs/2503.07465, repository https://github.com/THU-MIG/yoloe (AGPL-3.0)
• Example code and output: in this folder, code/12_open_vocab_head.py, code/12_open_vocab_head.out.txt