Status: Draft (2026-10-13), partially measured. Mask loss examples are values run directly on a laptop CPU (code/09_mask_losses.py, under 1 second). The original SAM was not run in our environment (unmeasured), and its speed is cited from the measured values of the successor SAM 2.1 tiny in Part 3 of the TT Score Model Anatomy. Paper figures are transcribed from the original text along with their conditions. Writing rule: Measured data is the default, and unmeasured items are marked as "Unmeasured".

Segment Anything

Table of Contents

• 1. Changing the Question of Segmentation
• 2. Architecture: Heavy Once, Light Multiple Times
  • Ambiguity: Producing Three Answers
• 3. Training Loss: Focal + Dice
• 4. Data Engine: The Model Creates Data, and Data Scales the Model
• 5. Limitations
• 6. Connection to Our Project
• References

1. Changing the Question of Segmentation

Segmentationis the task of painting "what this pixel belongs to" for every pixel. Like U-Net in Part 3, traditional segmentation models had what to paint predetermined during training. Models trained on cells paint only cells, and models trained on humans paint only humans.

SAM (Segment Anything Model) by Kirillov et al. changes the question. If a human gives a prompt(point clicks, boxes, approximate masks), it is a prompt segmentationtask that paints the object at that spot. "What" is determined at inference time rather than during training. Just as language models undergo large-scale training on a single task of predicting the next token and then perform various jobs via prompts, this is an attempt to create such a "foundation model" for segmentation.

2. Architecture: Heavy Once, Light Multiple Times

SAM consists of three parts.

1. Image Encoder: ViT-H/16 pre-trained with MAE (Part 7). It converts images resized to 1024×1024 into a 64×64 feature map (reduced 16x) and reduces channels to 256 using 1×1 and 3×3 convolutions. Due to high resolution, most layers perform attention only within 14×14 windows, with global attention restricted to 4 layers (N² problem in Part 7, Section 3).
2. Prompt Encoder: Converts point and box coordinates into small vectors using positional encoding. Approximate masks are reduced via convolution and added to the image features.
3. Mask Decoder: A lightweight 2-layer transformer. It generates masks after bidirectional cross-attention where prompt tokens look at image features (prompt → image) and image features look at prompt tokens (image → prompt).

The core of the design is computational asymmetry. The decoder consumes less than 1% of the encoder's computations, and if image features are pre-computed, the prompt encoder and mask decoder run in about 50ms on a web browser CPU , according to the paper. On a screen where a user modifies masks by changing clicks, the encoder needs to be run only once per image, and only the decoder needs to be re-run on each click.

Ambiguity: Producing Three Answers

A single point on a racket rubber is ambiguous: is it the "rubber," the "entire racket," or the "person holding the racket"? All three are plausible answers. If only a single output is forced, the model learns the average of the three—a blurry mask that represents none of them. SAM outputs three masksfor a single prompt along with their respective expected IoUs(the model's own score on how accurate the mask is). During training, only the one closest to the ground truth receives a loss penalty.

3. Training Loss: Focal + Dice

Mask loss mixes focal loss (Part 4) and dice loss in a 20:1 ratio. Dice scores based on the ratio of overlapping areas.

dice 손실 = 1 − 2·|예측 ∩ 정답| / (|예측| + |정답|)

Why not use only pixel average loss (BCE/focal)? We measured the loss of predictions where a small ball (81 pixels) and a large ping-pong table (29,885 pixels) in a 256×256 mask were each half-painted (measured).

Despite being the same "half-miss," the pixel average loss evaluates the ball's error as about 1/300th of the ping-pong table's. Dividing by the total pixels in the image dilutes errors for small objects. Because dice divides by the object's own area, it penalizes both similarly (0.45, 0.36). This is why dice is mixed into SAM, which must segment "anything" across vastly different sizes.

"Penalizing only the closest one" for ambiguous clicks was also verified in the same example. For the three outputs (rubber, racket, and person shapes), if the ground truth was the rubber, output 0 had the lowest loss; if the racket, output 1; if the person, output 2. Only that single output received the training signal. If a single output were produced and forced to match all three, the average would result in a blurry mask like 0.90 rubber, 0.60 handle, 0.30 body, which represents none of them (code/09_mask_losses.out.txt).

4. Data Engine: The Model Creates Data, and Data Scales the Model

To segment "anything," data where anything is segmented is required. Such data did not exist. The SAM team ran a data enginethat alternately scaled the model and the data.

Thus created SA-1B masks are 99.1% fully auto-generated. The paper explains that stage 2 took 34 seconds per mask again because the model filled in all the obvious ones, leaving only difficult, less noticeable objects for humans. This is a design that directs human time to areas where the model fails.

The average image resolution is 3300×4950, and the public release was reduced to a short side of 1500px. The model was trained for 68 hours on 256 datacenter GPUs (Model Card). Code and weights are under Apache-2.0, and SA-1B data is licensed for research.

Evaluation was performed in a "zero-shot" manner, producing masks from a single point across 23 segmentation datasets not used in training. The paper reports both automated metrics and human evaluations, but this post omits dataset-specific numbers.

5. Limitations

• Does not know names. SAM's masks do not contain "what this is." It merely segments the object at the clicked point. To search for "find the ping-pong table" using text, SAM 3 from Part 11 or open-vocabulary detectors from Part 12 are needed.
• The encoder is heavy. ViT-H 1024×1024 encoding is not lightweight unlike the decoder. It is difficult to run on every frame of real-time video (encoder speed was not verified in this post).
• Does not know video. Because it segments each frame separately, tracking the same object across the next frame requires providing prompts again. SAM 2 from Part 10 solved this.
• Ambiguity does not end at three. Hierarchies beyond the three ("rubber, racket, person"—such as just the handle of the racket, or both players) require additional prompts.

6. Connection to Our Project

In finding the ping-pong table in TT Score, the tap where a human points to the table directly becomes the prompt. Without needing to train "what to paint," a human just points once—this is precisely the prompt segmentation of this post. Moreover, thanks to the "one-time encoder, multi-decoder" architecture, modifying the tap does not take long to re-segment. The measured values for the successor SAM 2.1 tiny (server GPU, shared with other tasks) were 40ms for image encoding and 26ms for prompt prediction (TT Score Model Anatomy Part 3).

However, masks segmented with a single tap sometimes covered not only the table surfacebut also the side panels and legs, or only half the table due to the net blocking it. Therefore, we provide two taps (one for each half separated by the net) and re-fit the four straight edges from the mask to obtain the corners. Refining the model's output once more using geometry determined accuracy (Section 3 of the same post).

Ti-Ti's Word. Heavy encoder once per image, lightweight decoder per tap. That's why even if you re-point to the ping-pong table, it repaints instantly.

References

• Kirillov, A. et al. (2023). Segment Anything. ICCV 2023. https://arxiv.org/abs/2304.02643, repository https://github.com/facebookresearch/segment-anything (Apache-2.0)
• He, K. et al. (2022). Masked Autoencoders Are Scalable Vision Learners. CVPR 2022. https://arxiv.org/abs/2111.06377 (Pre-training of SAM encoder. Principles covered in sister series (Self-Supervised/Video/Pose/Sports))
• Milletari, F., Navab, N., Ahmadi, S.-A. (2016). V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation. 3DV 2016. https://arxiv.org/abs/1606.04797 (Dice loss)
• Our measurement: TT Score Model Anatomy Part 3 (SAM 2.1 tiny), repository docs/tt/PERFORMANCE.md 2.4
• Example code and output: In this folder code/09_mask_losses.py, code/09_mask_losses.out.txt