Status: Draft (2026-10-13), partially measured. Mask loss examples are values run directly on a laptop CPU (code/09_mask_losses.py, under 1 second). SAM (original) was not run in our environment (not measured), and its speed cites the actual measurement of the successor SAM 2.1 tiny from TT Score Model Anatomy Part 3. Paper figures were transcribed by opening the original text along with the conditions. Writing rules: Measured data is the default, and things that could not be measured are marked as "Not measured".

Segment Anything

Table of Contents

• 1. Changing the Question of Segmentation
• 2. Architecture: Heavy Once, Light Multiple Times
  • Ambiguity: Output Three Answers
• 3. Training Loss: focal + dice
• 4. Data Engine: The Model Creates Data, and Data Grows the Model
• 5. Limitations
• 6. Connection to Our Project
• References

1. Changing the Question of Segmentation

Segmentationis the task of painting "what this pixel belongs to" for every pixel. Like U-Net in Part 3, previous segmentation models had what to paint determined at training time. Models that learned cells paint only cells, and models that learned humans paint only humans.

SAM (Segment Anything Model) by Kirillov et al. changes the question. If a human gives a prompt(point click, box, approximate mask), it is prompt segmentationthat paints the object in that spot. "What" is decided at inference time, not training time. Just like language models perform large-scale training on a single task of next-token prediction and then do various tasks via prompts, this is an attempt to create such a "foundation model" for segmentation as well.

2. Architecture: Heavy Once, Light Multiple Times

SAM consists of three parts.

1. Image Encoder: ViT-H/16 pretrained with MAE (Part 7). It converts an image resized to 1024×1024 into a 64×64 feature map (reduced by 16x), and reduces channels to 256 using 1×1 and 3×3 convolutions. Because the resolution is large, most layers attend only within a 14×14 window, and global attention is kept in only 4 layers (N² problem in Part 7 Section 3).
2. Prompt Encoder: Converts point and box coordinates into small vectors using positional encoding. Approximate masks are reduced via convolution and added to the image features.
3. Mask Decoder: A lightweight 2-layer Transformer. It outputs masks after bidirectional cross-attention where prompt tokens look at image features (prompt → image) and image features look at prompt tokens (image → prompt).

The core of the design is computational asymmetry. The decoder uses less than 1% of the encoder's computation, and if image features are pre-computed, the prompt encoder + mask decoder runs in about 50ms on a web browser CPU as reported in the paper. On a screen where a user changes clicks to refine masks, the encoder only needs to run once per image, and only the decoder needs to be rerun with every click.

Ambiguity: Output Three Answers

A single point on a racket rubber could ambiguously mean the "rubber," the "entire racket," or the "person holding the racket." All three are plausible answers. If only a single output is produced, the model learns the average of the three answers—a blurry mask that is none of them. SAM outputs 3 masksand the respective expected IoU(the model's self-score on how accurate this mask will be) for a single prompt. During training, only the closest one to the ground truth receives loss.

3. Training Loss: focal + dice

Mask loss mixes focal loss (Part 4) and dice loss in a 20:1 ratio. Dice scores by the ratio of overlapping area.

dice 손실 = 1 − 2·|예측 ∩ 정답| / (|예측| + |정답|)

Why not use only pixel-average loss (BCE/focal)? We measured the loss of predictions where a small ball (81 pixels) and a large table tennis table (29,885 pixels) in a 256×256 mask were each painted halfway (measured).

虽然同样是“漏掉一半”，但像素平均损失将球的失误视为乒乓球桌的约 1/300。由于除以了整张图的像素，小物体失误的影响被稀释了。而 dice 是除以物体自身的面积，因此对两者的惩罚差不多（0.45、0.36）。这就是为什么在需要分割大小不一的“任何事物”的 SAM 中会混合使用 dice 的原因。

关于模糊点击的“只惩罚最近的一个”，我们在相同的示例中也得到了证实。对于输出集合（胶皮、球拍、人体形状），如果正确答案是胶皮，则是第 0 个输出；如果是球拍，则是第 1 个；如果是人，则是第 2 个输出的损失最小，并且只有该输出接收学习信号。如果只产生一个输出并试图迎合这三者，其平均值就会像胶皮 0.90、手柄 0.60、身体 0.30 那样，变成一个模糊不清、什么都不是的面具（code/09_mask_losses.out.txt).

4. 数据引擎：模型创造数据，数据培育模型

为了分割“任何事物”，我们需要准备好分割了任何事物的数据。但过去并没有这样的数据。SAM 团队运行了交替培育模型和数据的 数据引擎。

以此构建的 SA-1B 中， 99.1% 的掩码是完全自动生成的。论文解释说，在第 2 阶段单个掩码重新需要 34 秒的原因是，模型已经填满了所有显而易见的物体，只留下了对人类而言不那么明显的高难度物体。这一设计让人类的时间能够花在模型无法搞定的地方。

图片的平均分辨率为 3300×4950，公开发布版本将其缩减至短边 1500px。模型使用 256 张数据中心 GPU 训练了 68 小时（模型卡）。代码和权重采用 Apache-2.0 许可证，SA-1B 数据采用研究用途许可证。

评估采用“零样本”方式进行，在未用于训练的 23 个分割数据集上通过单个点输出掩码。论文同时报告了自动指标和人工评估，但在本文中未列出各个数据集的具体数字。

5. 局限性

• 不知道名称。 SAM 的掩码中没有“这是什么”的信息。它只是分割了点击位置的物体。如果要用文字实现“寻找乒乓球桌”，则需要第 11 篇 SAM 3 或第 12 篇开放词汇检测器。
• 编码器很重。 与解码器不同，ViT-H 1024×1024 编码并不轻量。很难在实时视频的所有帧上运行（本文未能确认编码器的速度）。
• 不懂视频。 由于对每一帧单独进行分割，如果要在下一帧中继续分割同一个物体，必须重新提供提示词。解决这一问题的是第 10 篇 SAM 2。
• 模糊性并非只有三种结果。 超出“胶皮·球拍·人”这三者的层级（例如仅球拍的手柄、两名选手等）需要提供更多提示词。

6. 与我们项目的联系

在 TT Score 的寻找乒乓球桌功能中， 人类点击乒乓球桌的标签直接成为了提示词。无需训练“该分割什么”，人类只需指出一次即可，这正是本文的提示词分割。此外，得益于“一次编码，多次解码”的架构，即使修改了标签，重新分割也不需要花费太长时间。后续版本 SAM 2.1 tiny 的实际测量值（在服务器 GPU 上、与其他任务共享时）显示，图像编码为 40ms，提示词预测为 26ms（《TT Score 模型解剖》第 3 篇）。

不过，仅通过一个标签分割出的掩码有时不仅会覆盖桌子的 表面，还会覆盖侧板和桌腿，或者被球网阻挡而只能分割出一半。因此，我们提供了两次标签（被球网隔开的两半各一个），并通过掩码中四条边的直线重新拟合来获取边缘。不直接使用模型分割的结果，而是用几何学再修饰一次，这决定了准确度（同篇文章第 3 节）。

TT 小结。 沉重的编码器每张图只需一次，轻量的解码器每次点击一次。因此，即使重新指定乒乓球桌，也能很快重新分割出来。

参考文献

• Kirillov, A. et al. (2023). Segment Anything. ICCV 2023. https://arxiv.org/abs/2304.02643, 代码库 https://github.com/facebookresearch/segment-anything (Apache-2.0)
• He, K. et al. (2022). Masked Autoencoders Are Scalable Vision Learners. CVPR 2022. https://arxiv.org/abs/2111.06377 (SAM 编码器的预训练。原理参见姊妹系列（自监督·视频·姿态·体育）)
• Milletari, F., Navab, N., Ahmadi, S.-A. (2016). V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation. 3DV 2016. https://arxiv.org/abs/1606.04797 (dice 损失)
• 我们的测量：《TT Score 模型解剖》第 3 篇 (SAM 2.1 tiny)，代码库 docs/tt/PERFORMANCE.md 2.4
• 示例代码与输出：该文件夹下的 code/09_mask_losses.py, code/09_mask_losses.out.txt