Contrastive Learning and CLIP: Learning by Pairing Images and Text to Guess Unseen Things

Status: Draft (2026-10-13). Paper figures are values confirmed with their conditions by opening the original source (arXiv). The only hands-on implementation is a toy example run on a laptop CPU (code code/01_infonce.py), and there is no actual implementation applying CLIP to table tennis videos (unmeasured). Series: "Reading Vision Models 2: Self-Learning, Motion, Humans, and Sports" Part 1. Basics such as convolutions, ViTs, detection, and SAM are covered in the sister series Reading Vision Models 1.

Table of Contents

• 1. Intuition: Matching Pairs Instead of Answer Keys
• 2. Illustration: A Single Batch Is a Single Workbook
• 3. How It Is Trained: InfoNCE
  • Loss Formula
  • Temperature τ
  • Creating Classifiers from Text
• 4. Hands-On Practice
  • Manually Calculated InfoNCE
  • Toy CLIP: Guessing Unseen Concepts
• 5. Paper Figures
  • SigLIP Converted to Sigmoid
• 6. Limitations
• 7. Connection to Our Project
• References

1. Intuition: Matching Pairs Instead of Answer Keys

Standard image classifiers learn from human-defined lists of labels, such as "This photo is a cat." They can never answer anything outside of the list. If there are 1,000 output slots, there is no space in the model for the 1,001st concept.

CLIP (Contrastive Language–Image Pre-training) changes the question. Instead of "What is the label of this image?", it asks, **"Which of these texts is paired with this image?"** On the internet, images and the texts describing them (captions, alt text) are frequently uploaded together. Pairs are created for free, and there is no upper limit to the variety of concepts.

Once it becomes good at matching pairs, classification comes as a bonus. By creating three sentences—"Photo of a table tennis racket," "Photo of a table tennis ball," and "Photo of a table tennis table"—we can check which of the three sentences is closest to a new image. This is true even though the label "table tennis racket" was never explicitly assigned during training. This is called zero-shot classification, meaning it solves tasks having seen zero examples.

2. Illustration: A Single Batch Is a Single Workbook

Training works as follows:

1. N image-text pairs are drawn as a single batch.
2. The image encoder (ResNet or ViT) converts N images, and the text encoder (Transformer) converts N texts into vectors of the same dimension, adjusting their lengths to 1 (unit vectors).
3. The inner product of all images and all texts, i.e., cosine similarity, is made into an N×N matrix. The diagonal represents true pairs, and the remaining N²−N cells represent fake pairs.
4. For each row, a classification problem of "selecting the diagonal cell" is solved, and the same problem is solved for each column. The average of the two losses is the final loss.

The key point is that there is no need to collect negatives separately. Other pairs in the same batch serve as negatives. The larger the batch, the more negatives are compared at once. CLIP set the batch size to 32,768 (according to the paper).

3. How It Is Trained: InfoNCE

Loss Formula

Letting the vector of image i be uᵢ, the vector of text j be vⱼ, and the temperature be τ, the image-side loss is as follows:

L_그림 = −(1/N) Σᵢ log [ exp(uᵢ·vᵢ / τ) / Σⱼ exp(uᵢ·vⱼ / τ) ]
L_글   = 같은 식을 열 방향으로
L      = (L_그림 + L_글) / 2

The inside of the parentheses is the softmax probability. It learns to push the probability of the correct answer cell closer to 1—that is, increasing the similarity of true pairs while decreasing the similarity of fake pairs. This form of loss is called InfoNCE . The name originates from the 2018 CPC (Contrastive Predictive Coding) paper. That paper showed that reducing this loss increases the lower bound of the mutual information between the two sides. Since the lower bound takes the form log N − L, the larger the number of negatives N, the tighter the lower bound becomes(paper). This is the theoretical reason for favoring large batches.

Temperature τ

τ is the value by which similarities are divided before being fed into softmax. When τ is small, even small differences in similarity widen significantly in probability (sharpness); when large, they get smoothed out. CLIP treated τ as a learnable parameter starting from a value corresponding to 0.07. Furthermore, it clipped the logit scale (1/τ) so that it would not exceed 100, which the paper notes was to prevent training instability.

Creating Classifiers from Text

Once training is complete, classification is performed as follows:

1. Sentences are created for each class name. Rather than just using the name itself, putting it into a template (prompt) like "A photo of a {label}." raised ImageNet accuracy by 1.3 percentage points. Creating 80 templates and averaging their vectors (prompt ensemble) raised it by an additional 3.5 percentage points (paper).
2. Sentence vectors act as the weights of the classifier. The text whose vector is closest to the new image's vector is the answer.

4. Hands-On Practice

code/01_infonce.py was run on a laptop CPU (numpy + PyTorch, around 20 seconds). The numbers are precisely from that output (code/01_infonce.out.txt).

Manually Calculated InfoNCE

).

logits = img @ txt.T / tau          # (N, N), 대각선이 정답 짝
loss = (ce(logits) + ce(logits.T)) / 2    # ce = 행마다 소프트맥스 교차 엔트로피(정답 = 대각선)

[[ 0.647  0.225  0.487  0.103]
 [ 0.333  0.459  0.425  0.286]
 [-0.255  0.302  0.445  0.267]
 [-0.383 -0.043 -0.318  0.351]]
τ=1.0   InfoNCE=1.1515   (무작위로 고르면 log N = 1.386)
τ=0.1   InfoNCE=0.4468
τ=0.07  InfoNCE=0.3604
τ=0.01  InfoNCE=0.5304

When τ=1, the difference between 0.65 and 0.49 is barely noticeable in terms of probability, making the loss almost indistinguishable from randomly chosen values (log 4 = 1.386). Lowering τ decreases the loss, but at 0.01 it rises again. Looking at the third column, the similarity with the first image (0.487) is greater than that of the correct answer (0.445). When the temperature is extremely low, it penalizes such a "wrong cell" very harshly. This is an example that intuitively demonstrates why τ was allowed to be learned rather than fixed.

Toy CLIP: Guessing Unseen Concepts

We constructed a dummy world consisting of 20 concepts. Each concept has a hidden vector, where "images" are 64-dimensional vectors and "texts" are 40-dimensional vectors produced by applying different non-linear transformations to that vector (including noise). The two encoders (small MLPs) are contrastively trained exclusively on pairs from concepts 0–14 , while concepts 15–19 are entirely excluded from training. During testing, 10 "texts" for each concept are encoded and averaged to use as prompts.

장난감 CLIP(배치 64, 600걸음) — 학습한 τ = 0.0756
  본 개념 15개 안에서 제로샷 정확도     : 0.973  (무작위 0.067)
  처음 보는 개념 5개 제로샷 정확도      : 0.855  (무작위 0.200)
지도 분류기(라벨 15개): 본 개념 정확도 0.973, 처음 보는 개념은 출력 칸이 없어 답할 수 없음

Training a 15-label classifier on the same data yields an identical 0.973 for seen concepts, but there are no output slots for new concepts. The contrastively trained model correctly guessed 85.5% of previously unseen concepts as well. Because it learned how to place images and text into the same space, having just the text of a new concept allows it to find its proper place.

We adjusted knobs to see what creates this capability (average of 3 seeds).

The most significant knob was the diversity of concepts used in training. Training with only 3 concepts causes the accuracy on new concepts to drop below half. Locking the temperature to 1 incurred a penalty of just over 7 percentage points. On the other hand, batch size showed no noticeable effect in this toy setting. We hypothesize this is because, with only 15 concepts, a large batch repeatedly includes the same concepts, turning those slots into false negatives that are "actually correct answers but treated as negatives" (estimated). Real CLIP, which uses 400 million distinct pairs, presents a different story. This result should not be directly transferred to real-world scale.

5. Paper Figures

All values are from the CLIP paper (Radford et al., 2021).

The final line is the takeaway of this post. Forcing models to correctly choose matching pairs is far cheaper to learn than having them generate sentences verbatim. While getting texts right word-by-word is difficult (there are countless sentences describing the same image), choosing pairs is easier while still teaching the meaning of the image.

SigLIP Converted to Sigmoid

SigLIP (Zhai et al., 2023) discards row and column softmax, applying independent binary classification per cell(1 if a pair, 0 otherwise) of the matrix using sigmoid. Since it does not normalize across the entire batch, there is no need to gather the entire similarity matrix across devices. To prevent initial collapse to "all zeros" due to the overwhelmingly large number of negative cells, it started with a bias of −10 and a scale of 10 (paper).

The observation that the conventional wisdom "larger batches are always better" halts around 32k aligns with the lack of batch size effects observed in our toy experiment above.

6. Limitations

• Weak at counting and fine-grained distinctions. The CLIP paper itself notes weaknesses in object counting, fine-grained classification such as car models, flower species, and aircraft types, and handwritten digits (MNIST, 88% zero-shot). MNIST scored even lower than a baseline running only logistic regression on pixels (paper).
• Fails to learn what text does not cover. Internet captions may say "person playing table tennis," but they never state "2 frames before the ball touches the receiver's table for the second time." Information that is difficult to describe in text, such as time and sequence, represents a blind spot of this approach (revisited in Part 12).
• Inherits data biases as-is. Because pairs are gathered from the web, web biases are naturally incorporated.

7. Connection to Our Project

• The roots of text-prompted segmentation and detection. Open-vocabulary detection and segmentation (such as SAM 3) that locate table tennis tables using the text "table tennis table" stand on this foundational idea of placing images and text in the same space. The structure is covered in the sister series Reading Vision Models 1Part 3 of the TT Score model dissection covers SAM and open-vocabulary detection, and how we applied it to the ping-pong table.
• The eyes of vision-language models. The LLaVA family, to be covered in Part 12, attaches a frozen CLIP image encoder to a language model. The fact that its eyes are weak at "counting" and "fine-grained timing" is part of why it struggles with sports officiating.
• Why we do not use it directly. TT Score's rulings require "how many, which side, and when," which hits squarely on the areas where CLIP is weak. Therefore, we did not include it in the scoring engine, and we have never even tested CLIP zero-shot on table tennis scenes (unmeasured).

A word from TT. Fascinating isn't it, how it only did "matching" without an answer key, yet gets things right even when seeing them for the first time? On the other hand, "how many?" or "when?" are not easily learned through matching alone. That is the job for our engine, which directly counts the ball.

References

• Radford et al., Learning Transferable Visual Models From Natural Language Supervision, ICML 2021. https://arxiv.org/abs/2103.00020
• van den Oord et al., Representation Learning with Contrastive Predictive Coding, arXiv 2018. https://arxiv.org/abs/1807.03748
• Zhai et al., Sigmoid Loss for Language Image Pre-Training(SigLIP), ICCV 2023. https://arxiv.org/abs/2303.15343
• Example code and output: in this series folder code/01_infonce.py, code/01_infonce.out.txt