SAM 3: Painting everything by calling concepts with text, and why we could not use it

Status: Draft (2026-10-13), unmeasured. SAM 3 could not be run in our environment (reasons in Section 6). The paper (arXiv v2), official repository, and Hugging Face page were opened on 2026-10-13 and transcribed along with their conditions, and hardware is noted by tier. The presence score calculation in Section 3 is a manual calculation for explanatory purposes, not a model output. Writing Rule: Measured data is the default, and unmeasured items are marked as "Unmeasured".

Table of Contents

• 1. From "This" to "All of These"
• 2. Architecture: Detector + Tracker, Single Encoder
  • Presence Token: Separating "Is it there?" and "Where is it?"
• 3. Data Engine: Humans + AI Verifier
• 4. Metrics and Scale
• 5. Limitations
• 6. Why We Could Not Use It
• References

1. From "This" to "All of These"

Parts 9 and 10 of SAM and SAM 2 receive visual prompts. Clicking a point paints "this." However, tasks like this are cumbersome to do with clicks.

• Painting all ping-pong balls in a stadium video (including balls rolling on the floor)
• Finding and painting "ping-pong tables" without clicking
• Painting all "rackets" across hundreds of images

SAM 3 by Carion et al. defines this task as Promptable Concept Segmentation(PCS). Given an image or short video (under 30 seconds) along with a noun phrase(short text like "white sneakers"), an example box(indicating what matches or does not match), or both, it finds and paints all objects corresponding to that concept and tracks them in videos. Existing click prompts (the SAM 2 approach) are also supported as-is.

2. Architecture: Detector + Tracker, Single Encoder

• Shared Vision Encoder: Using Meta's Perception Encoder as the backbone, the detector and tracker share the same features.
• Detector: Part 8's DETR family. Queries conditioned on text and example prompts each take charge of one object and output boxes and masks. Since "find all" is set prediction of an unknown count, DETR's set prediction fits well.
• Tracker: Inheriting Part 10's SAM 2 design, it follows detected objects in video memory.

Presence Token: Separating "Is it there?" and "Where is it?"

The new component in SAM 3 is the presence token. It is a global token that looks at the entire image and judges only whether "this concept exists in the image." Each query is responsible only for "where is it (location)," and the final score is the product of the two.

최종 점수 = 질의 점수(이 자리에 그것이 있다면 이 상자) × 존재 점수(그것이 그림에 있나)

Why separate them? A single query must look at only a part of the image and judge whether "this is a ping-pong table." A blue desk with white lines drawn on it looks like a ping-pong table when viewed only in part. On the other hand, "is there a ping-pong table in this image" is easier to judge using the context of the entire image (net, rackets, gym). Manual calculation looks like this.

(Explanatory numbers.) A desk that the query alone rated 0.60 as "probably a ping-pong table" is filtered out by being multiplied with the presence score. In the paper's ablation study, the presence head increased cgF1 by 1.5 points. cgF1 is a metric defined by this paper for PCS, evaluating both "is the concept present" judgments and per-object mask accuracy.

3. Data Engine: Humans + AI Verifier

Like SAM and SAM 2, SAM 3 also generated training data using a data engine. This time, an AI verifierhandles a portion of the human workload.

• Noun phrase candidates and easily confused "hard negative" concepts (e.g., "dining table" for "ping-pong table") are proposed by a Llama-family language model.
• Llama 3.2, fine-tuned on first-stage human labels, performs mask verification(does this mask match the concept) and completeness verification(are all objects of that concept painted). The paper reports that this roughly doubled the throughput of the data engine.

The original texts conflict in the last line. The paper (v1/v2) states 207Kunique concepts, while the official README states 270K . The human performance comparison is also 74% in the paper and 75–80% in the README. This article followed the paper. We could not verify which side is correct.

4. Metrics and Scale

The 848 million parameters are about 22 times those of SAM 2.1 tiny (38.9M). The comparison with Part 12's OWLv2 is because both perform "finding objects with text," and SAM 3 adds masks and video tracking on top of that.

5. Limitations

• Heavy. While 30ms for a single image on a datacenter GPU is fast, it is far from mobile phones or environments sharing small GPUs. For video, more objects require more GPUs.
• Ambiguity of Words. "Table" can mean a dining table or a ping-pong table. The shorter the noun phrase, the blurrier the concept boundaries. The feature of providing example boxes along with it mitigates this.
• Scope of Concepts. Things difficult to express with noun phrases (events like "the ball at the moment of serving") are outside the scope of this task.

6. Why We Could Not Use It

TT Score considered SAM 3 as a candidate for finding ping-pong tables without clicking, and even created code placeholders and alternative paths in other tasks (in progress). However, as of 2026-10-13, we could not run it. There are four reasons.

1. Weight Access Approval. Checkpoints require logging into Hugging Face, agreeing to share contact info, and applying for access to get approved (official README and model page). As of 2026-10-13, access was not granted for our token, so the hub rejected the download request (HTTP 403), and we could not obtain the weights.
2. Execution Environment. The official README requires Python 3.12 or higher, PyTorch 2.7 or higher, and CUDA 12.6 or higher. Our inference worker's Python is 3.11, so even with the weights, the current image gets blocked at the loading stage. We will revisit this when updating the image.
3. License. It is not Apache, but a separate "SAM License." It allows use, modification, and redistribution, but redistribution requires attaching the same agreement, use must be disclosed in papers, etc., reverse engineering is prohibited, military, weapon, nuclear, and intelligence uses and use subject to export controls (ITAR) are prohibited, a clause terminates the license if Meta is sued for intellectual property infringement, and a clause allows Meta to change terms. There is no non-commercial clause. To put it into a web service, these conditions must be reviewed.
4. Not Strictly Necessary. There is only one ping-pong table on the screen, and a person can point to it once. The advantage of "all things matching the concept" is small. We already achieved a median corner difference of 4.7px with geometry using SAM 2.1 tiny (TT Score Model Anatomy Part 3).

Therefore, for now, it remains a "candidate to be tested if approval is granted and the execution environment matches." If tested, we will measure the accuracy and latency of click-free finding in the same ping-pong table set as SAM 2.1 and append it to this article.

TT's Word. It's a model that finds everything when you ask in text, "Where is the ping-pong table?", but the door is locked. When the key (approval) arrives, we will look for our ping-pong table first.

References

• Carion, N. et al. (2026). SAM 3: Segment Anything with Concepts. ICLR 2026. https://arxiv.org/abs/2511.16719
• SAM 3 Repository (README, SAM License): https://github.com/facebookresearch/sam3
• SAM 3 Model Page (Access Request): https://huggingface.co/facebook/sam3
• Ravi, N. et al. (2025). SAM 2: Segment Anything in Images and Videos. ICLR 2025. https://arxiv.org/abs/2408.00714
• Carion, N. et al. (2020). End-to-End Object Detection with Transformers. ECCV 2020. https://arxiv.org/abs/2005.12872