Status: Draft (2026-10-13). Paper metrics are values verified directly from the original papers along with their conditions. Hands-on parts use a toy example on a notebook CPU (code/02_ssl_collapse.py, around 30 seconds), and we have not performed self-supervised learning on table tennis videos (unmeasured). Part 2 of the series "Reading Vision Models." Previous part: Contrastive Learning and CLIP. For the ViT architecture itself, see Reading Vision Models 1.

Self-Supervised Learning 1: Learning Features Without Labels

Table of Contents

• 1. Intuition: Labels Are Expensive, Images Are Abundant
• 2. The Problem: Collapse
• 3. How to Train
  • SimCLR: Large Batch and Strong Augmentations
  • MoCo: Storing Negative Examples in a Queue
  • BYOL: Surviving Without Negative Examples
  • DINO: Following the Teacher's Distribution
  • DINOv2: Data Over Methods
• 4. Hands-on: Seeing Collapse with Your Eyes
• 5. Paper Metrics
• 6. Limitations
• 7. Connection to Our Project
• References

1. Intuition: Labels Are Expensive, Images Are Abundant

Thinking about table tennis videos makes this immediately clear. Practice videos recorded with a smartphone pile up for hours, but manually labeling "the ball is here in this frame" yields at most a few hundred images an hour. Can we then learn something from the pile of unlabeled images itself?

Self-supervised learning creates tasks within the data and solves them. The problem family covered in this post works as follows:

Even if an image is augmented in two different ways (cropping, color jitter, blurring), the model must say they are the "same thing."

Features that survive augmentations are likely to capture the contentof the image while discarding its appearance(brightness, crop location). While CLIP in Part 1 paired "images and text," here we pair "an image and another augmented view of that image." Even text is unnecessary.

2. The Problem: Collapse

However, there is a catch. If "making the outputs of two augmented views identical" is the entire goal, a model that produces the exact same output for every inputis a perfect solution. The loss is at its minimum, but the features contain zero information. This is called collapse .

It is easy to summarize the methods in this post by viewing them all as different mechanisms to prevent collapse.

3. How to Train

SimCLR: Large Batch and Strong Augmentations

SimCLR's loss is called NT-Xent, but it has the same form as the InfoNCE loss from Part 1. From N images, 2N augmented views are created; the positive target for each view is its paired counterpart, and the remaining 2N−2 views are negative examples. The paper highlights three main points:

• The combination of augmentations is decisive. Using random cropping and color distortion together was especially crucial. Using only cropping causes the model to find pairs based solely on color distribution.
• Projection head. A small MLP is placed on top of the encoder output h to produce z, and the loss is applied to z. However, the features used later are h before the head. h outperformed z by over 10% in linear evaluation (paper). This suggests the projection head absorbs information that would otherwise be discarded to satisfy the loss.
• Large batch size. Default 4,096, experiments used 256–8,192. A temperature of 0.1 worked best (ImageNet 100 epochs, paper).

MoCo: Storing Negative Examples in a Queue

Large batches consume a lot of memory. MoCo retrieves negative examples from past batches. Outputs are stored in a queue of length 65,536 and used as negatives. The issue is that the outputs in the queue were produced by encoders from different timesteps. Therefore, the encoder filling the queue (the key encoder) does not backpropagate gradients, but instead tracks the weights of the trained encoder very slowly.

key 인코더 ← m · key 인코더 + (1 − m) · query 인코더,   m = 0.999

Setting m to 0 (copying directly every time) caused training to fail; it reached 55.2% at 0.9 and 59.0% at 0.999 (paper, ablation study with a queue of 4,096). Slowly tracking teacher, this mechanism carries over to BYOL and DINO.

BYOL: Surviving Without Negative Examples

BYOL does not use negative examples at all. A predictoris attached on top of the student (online) encoder to match the output of the target (teacher) encoder that has seen a different augmentation. The teacher is an EMA of the student and does not receive gradients (gradually increasing from 0.996 to 1). While the paper does not provide a complete mathematical proof for why the shortcut of "collapsing to a single point" is prevented, experiments clearly show that removing the predictor leads to collapse (paper). We can observe the same result in the example below.

DINO: Following the Teacher's Distribution

DINO turns outputs into probability distributions of dimension K. The teacher and student observe different crops of the same image, and cross-entropy is minimized so that the student matches the teacher's distribution. Because this resembles distilling a classifier, it is called self-distillation (distillation is covered in detail in Part 4).

# 02_ssl_collapse.py 의 DINO 부분에서 줄임
with torch.no_grad():
    q = softmax((teacher(v1) - center) / 0.04)         # 가운데 맞추고, 낮은 온도로 날카롭게
loss = -(q * log_softmax(student(v2) / 0.1)).sum(1).mean()
center = 0.9 * center + 0.1 * teacher_outputs.mean(0)    # 교사 출력의 이동 평균
teacher = EMA(student)

The two components that prevent collapse work in opposite directions:

• Centering: Subtracts the mean from the teacher output. This prevents the model from biasing toward a single dimension always yielding large values. However, using only centering causes outputs to collapse by spreading uniformly across all dimensions.
• Sharpening: Applies a low temperature (0.04–0.07) exclusively to the teacher side to make the distribution peaky. This prevents uniform spreading. Using only sharpening causes the model to collapse into a single dimension.

The paper showed through illustrations that using only one of the two causes collapse in its respective direction. The student views two large crops and multiple small crops, while the teacher views only large crops (multi-crop). This turns into the task of "predicting the meaning of the entire image from just small fragments."

DINOv2: Data Over Methods

DINOv2 did not significantly alter the loss (DINO loss + iBOT loss at the masked patch level + KoLeo regularization to spread features evenly). Instead, it focused heavily on data curation. From a deduplicated source of 1.2 billion images, it searched for images resembling established datasets to select 142 million images (LVD-142M). After training the largest ViT-g (1.1 billion parameters), smaller models were not trained from scratch but distilled from the large model(paper). Why distillation wins is the topic of Part 4.

4. Hands-on: Seeing Collapse with Your Eyes

code/02_ssl_collapse.py generates 32-dimensional "images" derived from 10 hidden classes. The images contain a highly fluctuating "background" in addition to the content defining the class, meaning that comparing raw pixels causes the background to dominate distances. Augmentations include repainting the background, erasing 30% of pixels, brightness scaling, and noise. All methods were trained using the same encoder (a small MLP) and the same 1,500 steps. Labels (class numbers) are used only for evaluation .

Evaluation consists of two metrics: the standard deviation per embedding dimension (near 0 means all points collapse to one), and k-NN accuracy, which polls the 5 nearest class numbers for each test image (using raw outputs, code/02_ssl_collapse.out.txt).

There are several takeaways here:

• Simply pulling features together yields a near-perfect loss, but the features are completely useless. This is the most dramatic example showing that a low loss does not guarantee a good model.
• BYOL's predictor and DINO's two components are critical load-bearing parts. Removing even one of them causes collapse, and the direction of collapse matches exactly what the DINO paper describes. Removing centering caused only 1 out of 32 dimensions to be used, while removing sharpening caused all images to output the same uniform distribution.
• In this toy setting, SimCLR performed best and DINO performed lowest. This does not indicate the absolute superiority of the methods. Hyperparameters (temperature, EMA speed, number of dimensions) were not tuned for the toy task, and DINO's strengths—multi-crop and ViT—were not used.

5. Paper Metrics

Evaluations are mostly linear evaluations. The self-supervised encoder is frozen, a single linear classifier is trained on top of it using ImageNet labels, and accuracy is measured. This checks how well-organized the features are so that they are "linearly separable."

There are also numbers regarding the importance of augmentations. When restricted to only cropping, BYOL maintained 59.4% (−13.1%p), whereas SimCLR collapsed to 40.3% (−27.6%p) (BYOL paper). This suggests that methods relying on negative examples are more vulnerable to shortcuts like "finding pairs based solely on color."

Besides the metrics, DINO has another famous observation. Without ever seeing labels, the attention maps in the final layer of the ViT trace the outlines of objects. The mask overlap (Jaccard) created using these maps was 45.9 for ViT-S/16, which is higher than supervised training with the same architecture (27.3) (Paper).

The computation cost is as follows: DINO ViT-S/16 took 3 days on two 8-GPU servers, the entire DINOv2 took 22,000 datacenter-class GPU hours, and DINOv3 took 61,000 hours (Paper).

6. Limitations

• Augmentations are assumptions. If you teach a model that "even if the color changes, it is the same thing," the model discards color. If you need to distinguish between an orange ball and a white ball in table tennis, color augmentation works to erase that information.
• Tailored to single-image semantics. This family of models is strong at "what is this image?" Information like position, such as where the ball is, or time, such as when it bounced, requires separate mechanisms (mask-and-predict in Part 3, video models in Part 6). DINOv3's Gram anchoring is a mechanism introduced to address the issue of dense patch-level features breaking down during long training sessions (Paper).
• Larger and longer. Good numbers typically came from hundreds to thousands of epochs and large datasets. There is no guarantee that the same effects will be achieved with smaller datasets.

7. Connection to Our Project

• Where labels are most precious. Most of the training data for TT Score are publicly available datasets under non-commercial conditions, and our mobile phone videos have almost no labels (TT Score Model Anatomy Part 7). Performing self-supervised learning first on our unlabeled videos and then fine-tuning with a small number of labels is a direction well-suited for this problem. However, we have not tried this yet, and it remains unknown (unmeasured) how much this family of models will help with small objects like balls.
• Slow teacher. MoCo's key encoder and BYOL/DINO's EMA teachers share the same idea that "the student's past average becomes a more stable teacher than the student." We will revisit this in Part 4 (Distillation) and Part 6 (V-JEPA).
• Borrowing features. Using a frozen public feature extractor like DINOv2 can allow us to start with fewer labels to segment the table tennis table area or player areas (estimated, unmeasured).

TT's Word. If you tell the model, "Just say the two are the same," the model ends up saying everything in the world is the same. It was fascinating to see firsthand that removing even one mechanism preventing that—whether it's negative examples, a predictor, or centering—truly causes it to collapse.

References

• Chen et al., A Simple Framework for Contrastive Learning of Visual Representations(SimCLR), ICML 2020. https://arxiv.org/abs/2002.05709
• He et al., Momentum Contrast for Unsupervised Visual Representation Learning(MoCo), CVPR 2020. https://arxiv.org/abs/1911.05722
• Chen et al., Improved Baselines with Momentum Contrastive Learning(MoCo v2), arXiv 2020. https://arxiv.org/abs/2003.04297
• Grill et al., Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning(BYOL), NeurIPS 2020. https://arxiv.org/abs/2006.07733
• Caron et al., Emerging Properties in Self-Supervised Vision Transformers(DINO), ICCV 2021. https://arxiv.org/abs/2104.14294
• Oquab et al., DINOv2: Learning Robust Visual Features without Supervision, TMLR 2024. https://arxiv.org/abs/2304.07193
• Siméoni et al., DINOv3, arXiv 2025. https://arxiv.org/abs/2508.10104
• Example code and output: code/02_ssl_collapse.py, code/02_ssl_collapse.out.txt