Status: Draft (2026-10-13). Paper figures were verified by opening the original text along with their conditions. Hands-on work is a toy example on a laptop CPU (code/03_mae_toy.py, around 3 minutes). We have never trained MAE with actual photos or table tennis videos (unmeasured). Part 3 of the series 「Reading Vision Models 2」. Previous part: Self-Supervised Learning 1. For how ViT divides images into patches, see the ViT part of Reading Vision Models 1.

Self-Supervised Learning 2: MAE

Table of Contents

• 1. Intuition: Fill-in-the-Blank
• 2. Image: Asymmetric Encoder-Decoder
• 3. How It Is Trained
  • Loss
  • Labels and Data
  • Masking Strategy
  • Difference from BEiT
• 4. Trying It Yourself
• 5. Paper Numbers
• 6. Limitations
• 7. Connection to Our Project
• References

1. Intuition: Fill-in-the-Blank

The methods in Part 2 taught us that "two variants are the same." MAE (Masked Autoencoder) poses a different problem: Mask a part of the image, and guess the masked part.

This problem has already achieved great success in text. BERT masked 15% of sentence words and had models guess them to establish the foundation of language models. We can apply the same idea to images. To plausibly fill in a masked area, you need to understand structures such as "the edge of the ping-pong table will continue here" or "this arm is attached to the shoulder."

However, masking only 15% of an image is too easy. It can be filled by roughly connecting neighboring pixels. Images have much less information densitythan text because neighboring pixels have almost the same values. MAE's answer is bold: it masks 75%. At this scale, simple connection doesn't work, and you must understand the structure of the entire image to fill it in.

2. Image: Asymmetric Encoder-Decoder

MAE's second idea is computation.

1. Images are divided into small square patches like ViT (16×16 pixels, 14×14 for ViT-H).
2. 75% of the patches are randomly selected. The selected patches are not fed into the encoder at all. The large ViT encoder processes only the visible 25%.
3. The encoder's outputs are put back in place, and empty slots are filled with a learnable vector (mask token) indicating "this part is masked."
4. A small, shallow decoder reconstructs the pixels of the masked patches.
5. Once training is complete, the decoder is discarded and only the encoder is used.

Transformer computation scales proportionally to the number of tokens (MLP) or quadratically with it (attention). Reducing tokens to a quarter significantly cuts the computation of the large encoder. The decoder was made shallow so that its computation per token is less than 10% of the encoder's (paper).

3. How It Is Trained

Loss

L = 가린 조각들에 대해서만 평균한 (예측 화소 − 정답 화소)²

Reconstruction errors for visible patches are not counted because copying visible things has nothing to learn.

One tweak is applied to target pixels: pixels with the mean subtracted and divided by the standard deviation for each patchare used as the target. This forces the model to focus on guessing patterns within the patch (edges, textures) rather than the overall brightness of the patch. In ViT-L, fine-tuning accuracy increased from 84.9% to 85.4%, and linear evaluation increased from 73.5% to 73.9% (paper).

Labels and Data

No labels are used. The paper used only ImageNet-1K images, followed by fine-tuning with labels or linear evaluation after pre-training.

Masking Strategy

Even with the same 75%, how it was masked mattered (paper, ViT-L, fine-tuning / linear evaluation).

Grid masking spreads visible patches evenly, making it easy for neighbors to connect, while masking 75% in large blocks makes it too difficult. Random masking provides a balance in between.

Difference from BEiT

BEiT, which came slightly before MAE, had models guess word numbers assigned to patches by a pre-trained tokenizer (dVAE, 8,192 words) instead of pixels. It masked about 40% in blocks (paper). MAE showed that pixels can be guessed directly without a tokenizer, resulting in a simpler structure.

4. Trying It Yourself

code/03_mae_toy.py draws two dots on a 16×16 grayscale image. The "pattern" (label) of the image is the arrangement of the two dots: horizontal side-by-side, vertical side-by-side, '\' diagonal, '/' diagonal. The position, size, and brightness of the dots vary per image. To understand the pattern, you must look at relationships between patches, not just a single patch.

We divided the image into 16 patches of 4×4 and trained MAE using a small transformer (3 encoder layers, 1 decoder layer, width 64). We trained for 800 steps changing only the masking ratio, attached a linear classifier to the encoder features, and measured pattern classification accuracy (linear probe). Pattern labels are used only for this probe.

# 03_mae_toy.py 에서 줄임
order = torch.rand(n, 16).argsort(1)             # 그림마다 다른 무작위 가리기
keep, masked = order[:, :n_keep], order[:, n_keep:]
z = encoder(gather(embed(patches) + pos, keep))  # 보이는 조각만 인코더로
full = mask_token.repeat(n, 16, 1); full.scatter_(1, keep, z)
pred = head(decoder(full + dec_pos))
loss = mse_per_patch(pred, normalize_per_patch(patches)).gather(1, masked).mean()   # 가린 조각에서만

Results (average of 2 seeds, code/03_mae_toy.out.txt):

The encoder trained solely by masking and guessing yielded much better features at any ratio than the untrained encoder (0.575) and raw pixels (0.407). This means it learned "relationships" without labels.

However, for this toy, 75% was not the best. 25–50% was slightly better (the difference between seeds was ±3%p, so 25% and 50% cannot be distinguished). This inversely shows MAE's core assumption. Our image has only 16 patches with mostly empty backgrounds, so information is concentrated in the two dots. Showing only 4 patches often causes both dots to disappear entirely, leaving no basis for guessing. High masking ratios are a prescription that works when images are full of redundancy. A single real photo has hundreds of patches, and neighboring patches resemble each other closely. In video (Part 6), neighboring frames resemble each other even more, pushing the ratio up to 90%.

Training time varied greatly depending on machine load. Running the same code twice, 25% took 21.5/21.6 seconds, 75% took 27.9/16.5 seconds, and 90% took 23.7/15.7 seconds (accuracy and error numbers were identical both times). Even in the second values run under less load, the effect of reducing encoder tokens to ¼ (12 to 4) only shortened time by about 25%. For a model of this size, fixed costs of Python and libraries, along with the unreduced decoder (all 16 tokens), dominate over token count. Verify the 3x+ gains in large VTs via the paper figures below.

5. Paper Numbers

All values are from the MAE paper (He et al., CVPR 2022) and were pre-trained using only ImageNet-1K.

The second row is interesting. MAE is weaker than contrastive learning in linear evaluation, but stronger in fine-tuning. This means features learned by masking and guessing are less suited for direct linear probing, but serve as an excellent starting point when slightly adapted. This is why the DINO family from Part 2 and the MAE family from this part have different strengths. ViTPose in Part 8 attaches a small decoder to an MAE pre-trained ViT to predict poses.

6. Limitations

• Effort spent on guessing pixels. Loss decreases only when details unrelated to semantics, such as leaf textures or background patterns, are also guessed. This led to follow-up research saying, "Let's guess features instead of pixels" (V-JEPA in Part 6 is an example).
• Weak linear evaluation. If you intend to freeze features and use them as-is, the family from Part 2 is better (table above).
• Weak basis for small objects. When 75% is randomly masked, objects consisting of only a few pixels, like a ping-pong ball, are usually masked entirely or visible entirely. The paper did not address how much MAE pre-training helps with such objects (unverified).

7. Connection to Our Project

• Video fits better. Mobile phone videos have nearly identical neighboring frames with massive redundancy. MAE's assumptions fit video better than images. This is why VideoMAE in Part 6 masks 90%.
• Foundation for pose models. ViTPose in Part 8 uses MAE pre-trained ViT as-is. Since this differs from RTMPose used in TT Score's pose coaching, it provides a basis for comparison between the two paths (TT Score Model Anatomy Part 4).
• Our unlabelled video. For the same reason as Part 2, the ability to use unlabeled practice videos is attractive. We haven't tried it yet (unmeasured).

Titi's Word. Being able to guess even when three-quarters are covered means there's a lot of repetition in the image. In a drawing with just two dots like my toy, covering three-quarters makes the dots disappear so you can't guess. The masking ratio must be chosen to match how repetitive the data is.

References

• He et al., Masked Autoencoders Are Scalable Vision Learners(MAE), CVPR 2022. https://arxiv.org/abs/2111.06377
• Bao et al., BEiT: BERT Pre-Training of Image Transformers, ICLR 2022. https://arxiv.org/abs/2106.08254
• Example code and output: code/03_mae_toy.py, code/03_mae_toy.out.txt