Vision Transformer: Reading an Image Like a Sentence of 16×16 Patches, and Why It Needs So Much Data

Status: Draft (2026-10-13), partially measured. Examples were run directly on a laptop CPU(code/07_vit.py, under 1 second). Paper figures were taken from the original sources along with their conditions, and hardware is noted by tier. Writing rule: Measured data is the default, and unmeasurable items are marked as 「unmeasured」.

Table of Contents

• 1. Attention: Looking at All Pairwise Relationships
• 2. ViT: Images as Sentences of Patches
  • Try it yourself 1: Patches + Linear Projection = Convolution with Stride 16
  • Try it yourself 2: Without Positional Embeddings, the Order is Unknown
• 3. The Cost of Resolution
• 4. Why So Much Data Is Needed
• 5. DeiT: Training with ImageNet Only
• 6. Returning to Hierarchical Structures: Swin, Hiera
• 7. Limitations and Our Project
• References

1. Attention: Looking at All Pairwise Relationships

The Transformer (2017) was created for machine translation. Its core operation is self-attention. For each word in a sentence, it generates three vectors.

• Query (Q): 「What am I looking for?」
• Key (K): 「What information do I hold?」
• Value (V): 「If you pick me, this is what I give」

It takes the dot product of the query of word i and the key of word j to determine 「how much i should look at j」, and mixes the values using that ratio.

Attention(Q, K, V) = softmax( Q·Kᵀ / √d_k ) · V

The paper explains the reason for dividing by √d_k as follows: When the dimension d_k is large, dot products become large (if components have mean 0 and variance 1, the variance of the dot product is d_k), causing softmax to peak on one side and gradients to become extremely small. Dividing scales the variance back down to around 1.

Compared to convolutions, the difference is clear. A single output pixel of a convolution looks only at a 3×3 neighborhood, and stacking layers was required to see farther (Receptive Field in Part 1). Attention looks at all positions from the very first layer. Who to look at is determined not by a fixed window, but by the data (Q and K).

2. ViT: Images as Sentences of Patches

ViT by Dosovitskiy et al. makes images look like sentences.

1. It cuts a 224×224 image into 16×16 patches. 14×14 = 196 patches.
2. A single patch (16×16×3 = 768 numbers) is flattened and converted into a D-dimensional vector (token) via linear projection.
3. A learnable CLS tokenis prepended. In the final layer, the output of this token is used as the representation of the entire image for classification.
4. A positional embedding(a learnable vector) is added to each token.
5. A standard Transformer encoder (self-attention + MLP, each with residual connections and layer normalization) is stacked for L layers.

The 「/16」 and 「/14」 in the names refer to the patch size. The image encoder used by SAM in Part 9 is this ViT-Huge (patch size 16).

Try it yourself 1: Patches + Linear Projection = Convolution with Stride 16

Cutting into patches and multiplying by a matrix is the same computation as sliding 16×16 filters with a stride of 16. We calculated and compared both methods using the same weights (measured).

224×224 → 패치 196개(14×14), 패치 하나 768개 숫자 → 64차원 토큰
「잘라서 선형 변환」과 「보폭 16 합성곱」의 최대 차 2.7e-15

The only difference is floating-point rounding. Most practical implementations also build this part as a single convolutional layer. The first layer of ViT is 「a very large convolution with non-overlapping windows」, and that is where the convolution-like nature ends.

Try it yourself 2: Without Positional Embeddings, the Order is Unknown

Self-attention does not care about the order of tokens. Shuffling the input tokens simply shuffles the outputs in the exact same way (permutation equivariance). Consequently, the model cannot tell if the top-left and bottom-right patches of an image are swapped (measured).

토큰 순서를 섞어 넣은 출력 == 원래 출력을 같은 순서로 섞은 것: 최대 차 1.3e-15
위치 임베딩을 더하면 같은 패치를 섞었을 때 출력이 달라진다: 상대 차 0.28

This is why positional embeddings are necessary. In the ViT paper's experiments (ViT-B/16, linear evaluation on 5 images per class from ImageNet), the results were 0.614 without positional embeddings, 0.642 with 1D learned embeddings, 0.640 with 2D embeddings, and 0.640 with relative positions. Having them or not makes a huge difference, but the specific type hardly matters. This means that at the patch level (14×14), providing just 1D indices allows the model to learn the 2D layout on its own.

3. The Cost of Resolution

Because attention looks at all pairs of tokens, its cost scales quadratically with the number of tokens N(measured computation).

SAM takes 1024×1024 inputs. Therefore, SAM's ViT-H performs attention only within 14×14 windows for most layers, and leaves global attention to only 4 layers (Part 9). Such compromises are necessary when using ViT for tasks where resolution matters.

4. Why So Much Data Is Needed

The most important result of the ViT paper is not the accuracy numbers, but the reversal in performance based on data size.

• When pre-trained solely on ImageNet-1k (approx. 1.3 million images), large ViTs underperformed ResNets (BiT), and ViT-Large even underperformed ViT-Base.
• On ImageNet-21k (approx. 14 million images), Large and Base performed similarly.
• Only on JFT-300M (approx. 300 million images) did the benefits of large models fully emerge, allowing ViT to surpass ResNet.

(Paper Table 2. ViT-H/14 fine-tuned at 518px, ViT-L/16 at 512px.)

The paper explains the reason as inductive bias. Convolutions are born with built-in structural priors: 「nearby pixels are closely related (locality)」 and 「the same pattern is the same everywhere (translation equivariance)」 (Part 1). In ViT, the only component possessing such properties is the MLP within patches, while self-attention is global. Consequently, the rule 「look more closely at nearby patches」 must be learned from data by the model itself , and when data is scarce, the model memorizes the training data before learning that rule. When data is sufficient, conversely, this freedom becomes an advantage. Because the architecture does not impose constraints, the model can discover better rules suited to the data.

Looking closely at a trained ViT reveals that even in the lowest layers , some heads look at most of the image while others look only at nearby regions, and the viewing distance increases as the layers go deeper (Section 4.5 of the paper). In effect, it has autonomously generated heads that look locally, much like convolutions.

The cost side is also interesting. In the table above, ViT-H/14 achieved higher accuracy than BiT-L while its pre-training cost was about 1/4. Given sufficient data, Transformers also proved advantageous in computational efficiency.

5. DeiT: Training with ImageNet Only

Most people cannot access 300 million private images. DeiT (2021) by Touvron et al. demonstrated how to successfully train ViT using ImageNet-1k alone . The key lies in two things.

1. Strong augmentations and regularization. RandAugment, mixup, CutMix, random erasing, stochastic depth, and repeated augmentation are all used. In an ablation experiment removing them one by one (DeiT-B, 224px), removing mixup and CutMix together dropped accuracy from 81.8% to 75.8%, removing repeated augmentation dropped it to 76.5%, and removing random erasing or stochastic depth caused training itself to fail. This is akin to teaching the inductive bias—which should be learned from data—by artificially augmenting the data.
2. Distillation token. Alongside the CLS token, an additional token is introduced to 「imitate the teacher model's answers」. The teacher is a convolutional network (RegNetY-16GF, 82.9%). This can be viewed as inheriting the convolutional inductive bias through the teacher's answers.

Training took 2–3 days on an 8-GPU server (53 hours for 300 epochs). Elsewhere in the paper, the phrase 「3 days with 4 GPUs」 also appears, showing conflicting conditions within the original text.

6. Returning to Hierarchical Structures: Swin, Hiera

ViT maintains a single 14×14 resolution from start to finish. For tasks like detection and segmentation that handle objects of various sizes, the pyramidstructure of convolutions (high-resolution to low-resolution) was missed.

• Swin Transformer (2021): Performs attention only within small windows, and shifts the windows by half a patch across layers to mix information between windows. The cost becomes linearwith respect to image size, and tokens are merged at each stage to form a pyramid. It achieved a COCO test-dev box AP of 58.7 and an ADE20K segmentation mIoU of 53.5.
• Hiera (2023): Stripped away the auxiliary components attached to hierarchical ViTs (relative positional embeddings, convolutions, overlapping pooling, etc.) and instead used strong MAE pre-training(masking and reconstructing parts of the image) to learn those roles. It became faster while maintaining accuracy (Hiera-L achieved about 2.4x higher throughput per second compared to the baseline MViTv2-L on datacenter GPUs with FP16). Hiera-H achieves 86.9% on ImageNet-1k. The image encoder of SAM 2 is Hiera (Part 10).

The principles of self-supervised pre-training like MAE are covered in the companion series (Self-Supervised Learning, Video, Pose, Sports).

7. Limitations and Our Project

• Data and compute. Because its inductive bias is weak, it requires massive data, strong augmentations, or pre-training.
• Resolution cost. Due to N² costs, high-resolution tasks require compromises such as windowed attention.
• Small objects. Objects smaller than a single 16×16 patch get swallowed within a single token. A ping-pong ball with a 3px diameter in model inputs occupies less than 1% of a patch.

Consequently, roles are split in TT Score: finding the ball uses convolutions (HRNet family), , and segmenting the ping-pong table uses a Transformer (Hiera in SAM 2.1). The table is large, simple, and only needs to be located once at the start of the match, making it acceptable to use a heavy Transformer; the ball is small and must be found in every frame, making a lightweight, local convolution appropriate (TT Score Model Anatomy Part 1).

The following post is about DETR, which rewrites detection with transformers. There are no anchors and no NMS.

TT's Word. Even if you shuffle the order of the patches, the model won't notice without positional embeddings. It's like looking at an image as a pile of puzzle pieces. You have to attach ID tags so the puzzle can be assembled.

References

• Vaswani, A. et al. (2017). Attention Is All You Need. NeurIPS 2017. https://arxiv.org/abs/1706.03762
• Dosovitskiy, A. et al. (2021). An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. ICLR 2021. https://arxiv.org/abs/2010.11929
• Touvron, H. et al. (2021). Training data-efficient image transformers & distillation through attention. ICML 2021. https://arxiv.org/abs/2012.12877
• Liu, Z. et al. (2021). Swin Transformer: Hierarchical Vision Transformer using Shifted Windows. ICCV 2021. https://arxiv.org/abs/2103.14030
• Ryali, C. et al. (2023). Hiera: A Hierarchical Vision Transformer without the Bells-and-Whistles. ICML 2023. https://arxiv.org/abs/2306.00989
• Example code and output: In this folder, code/07_vit.py, code/07_vit.out.txt