SAM 2 and EfficientTAM: Tracking Objects in Video with Memory Attention and Making It Lightweight

Status: Draft (2026-10-13), partial measurements added (2026-10-15). The speed and accuracy of SAM 2.1 tiny cite measurements from Part 3 of the TT Score Model Anatomy (2026-10-11–12, during a period of shared server GPUs and temporary CPU pods). The memory reduction example was run directly on a laptop CPU (code/10_memory_pool.py, a few seconds). Added on 2026-10-15: For EfficientTAM, which previously lacked measurements, Section 6 cites values measured under the same conditions as SAM 2.1 tiny for an automatic table tennis table detection task. The actual smoothness of the memory map remains unmeasured. Paper figures were transcribed from the original text along with their conditions, and hardware is specified by tier. Writing Rule: Measured data is the baseline, and unmeasured items are marked as "Unmeasured".

Table of Contents

• 1. From Images to Video: What Else Is Needed
• 2. Architecture: Remembering While Streaming
• 3. Data and Metrics
  • Our Measurements: SAM 2.1 tiny
• 4. EfficientTAM: Reducing Memory for Lighter Weight
  • Hands-On: Is Smoothness Really Enough?
  • Paper Figures and Points to Note When Reading
• 5. Limitations
• 6. Connection to Our Project
• References

1. From Images to Video: What Else Is Needed

Part 9's SAM segments a single image. To achieve "segment the object pointed to in the first frame all the way to the end" in a video, two more things are needed:

1. Memory: What the object looked like in previous frames. The model must recognize that it is the same object even if it rotates or is temporarily occluded.
2. Speed: Running a heavy encoder from scratch for every frame is slow.

SAM 2 by Ravi et al. solves both problems by attaching memoryto SAM's architecture. Because an image is treated as a "single-frame video," the same model segments both images and videos.

2. Architecture: Remembering While Streaming

Frames arrive one by one in sequence (streaming). Here is what happens in each frame:

1. Image Encoder: MAE-pretrained Hiera(hierarchical transformer from Part 7, Section 6) turns the frame into features. This runs once per frame.
2. Memory Attention: The current frame's features first attend to themselves (self-attention) and then to the memories in the memory bank (cross-attention). There are typically 4 such blocks.
3. Mask Decoder: A decoder identical to SAM generates the mask. If a user provided a click for the current frame, it is also used as a prompt. A separate prediction head determines whetherthe object is visible in this frame, allowing it to output "none" for completely occluded frames.
4. Memory Encoder: Downsamples the newly generated mask, adds it to the current frame's features, and blends them with lightweight convolutions to create the memory that "the object looked like this in this frame."
5. Memory Bank: Maintains separate queues for recent frame memories (typically 6 frames) and user-prompted frame memories (FIFO). It also stores object pointersextracted from the mask decoder's output tokens (256 dimensions split into four 64-dimension tokens). While the spatial map answers "where and how," the pointer summarizes "what the object is."

Intuitively, this means "find the object in the current frame by referencing both how it appeared in the last few frames and what it looked like in the scene where the user first pointed at it." Keeping the memory of the user-indicated frame for a long time is a crucial design choice. Relying solely on recent memories easily leads to drift, where slightly erroneous masks become memories that make subsequent frames even more inaccurate. The user-verified memory acts as an anchor.

3. Data and Metrics

SA-V Data: 50,900 videos, 642,600 masklets (per-frame mask bundles of an object in a video; 190,900 human + 451,700 automatic), and 35.5 million masks. Similar to SAM's data engine (Part 9), this dataset was built by running a model-segmentation and human-correction loop on videos.

SAM 2.1 checkpoints released in September 2024 (single datacenter GPU, all components compiled. Measurement environment differs from the paper). Size

Our Measurements: SAM 2.1 tiny

TT Score experimented with the smallest tiny model for table tennis table segmentation (TT Score Model Anatomy Part 3).

Condition

4. EfficientTAM: Reducing Memory for Lighter Weight

EfficientTAM by Xiong et al. aims to make SAM 2 lightweight enough to run on mobile phones. Two main changes were made:

Flat ViT Encoder.

1. Instead of the hierarchical Hiera, it uses a smaller single-resolution ViT (Tiny/Small, patch size 16), initialized via EfficientSAM's SAMI pretraining. Coarse Memory Cross-Attention.
2. Spatial tokens (keys and values) in memory are averaged over 2x2 windows to reduce them to one-fourth before attention is applied. Object pointer tokens are not reduced.

The rationale for item 2 is local smoothness. Since features of neighboring cells in a memory map are similar, replacing four cells with their average does not drastically change the attention output. The paper showed that in one example, the relative error compared to full attention was 0.03 (Figure 3, single example), and reported that further reducing it to 4x4 drops accuracy by about 1 point while yielding minimal speed gains.

Hands-On: Is Smoothness Really Enough?

We tested this assumption directly. Memory = 32x32 cells x 4 frames x 64 dimensions, 1,024 queries. We measured how much the attention output changes when keys and values are averaged in 2x2 blocks for two types of memory (measured values, laptop CPU, median of 7 runs).

Execution time decreased to about one-fourth, proportional to the key count. The error was small only when the memory was smooth. EfficientTAM's reduction is not a free lunch; it relies on the assumption that "video features are similar among neighbors", and is accurate only to the extent that this assumption holds. How smooth actual features are varies by model and scene, and we did not measure this using actual memory maps (Unmeasured).

Paper Figures and Points to Note When Reading

(Paper Table 1, 1024x1024 input. "/2" denotes the version where memory is reduced via 2x2. Mobile phone refers to a 2023 high-end device.)

A close look at the original text reveals discrepancies in two places.

• The size of the compared SAM 2 model is ambiguous. The SAM 2 model in the table has 81 million parameters (base+ size), but the main text also states that SAM 2 tiny achieves 43.8 FPS. Comparing this with the tiny model in the official SAM 2.1 table above (38.9 million parameters, SA-V 76.5) paints a different picture.
• The summary "mobile phone around 10 FPS" conflicts with the table. Table 1 shows 261–1,011ms per frame (1–4 FPS) for 1024x1024 input, and figures close to 10 FPS are found in a different table with 512x512 input (EfficientTAM-S 80.6ms).

Copying only numbers from a paper summary omits such conditions, which is why examining the original tables directly is necessary. Code and checkpoints are under Apache-2.0, and conference publication status has not been verified (arXiv version).

5. Limitations

• Drift and Long Occlusion. Memory consists only of the last few frames and prompted frames. If an object is occluded for a long time or a similar object (e.g., a neighboring table) passes by, the model may track the wrong thing.
• Small and Fast Objects. When tracking a ball with SAM 2 in TT Score, the recall for a ball overlapping with a player increased from 0.879 to 0.926, but the boundary precision dropped, resulting in zero accuracy gain for rally winners. Therefore, it was omitted (same Part 3).
• The Cost of Lightweight Variants. Flat encoders and coarse memories sacrifice a bit of accuracy (the Ti version is 4 points lower in the table above).

6. Connection to Our Project

• When the camera is fixed on a tripod, the table tennis table does not move. Therefore, rather than SAM 2's tracking, TT Score uses this model for painting once at the start of the match. It is fine if it takes a few hundred ms, and even on a CPU, it takes just over a second.
• As the feature to find the table tennis table without user taps and follow a shaking camera was added on 2026-10-13, we benchmarked EfficientTAM under the same conditions as SAM 2.1 tiny (103 images with ground-truth vertices × seven types of synthetic degradation = 721 images, 512×288, when the server GPU was shared with other tasks). The median vertex errors found without taps were 23.1px for SAM 2.1 tiny, 8.8px for EfficientTAM S, and 8.8px for EfficientTAM Ti, and the ratios of falling within 20px were 45% · 57% · 53%. A single detection took 1.48 · 1.65 · 1.55 seconds, which were similar, but most of the time was spent on CPU post-processing converting masks to vertices rather than the model. The additional GPU memory was 0.70 · 0.64 · 0.54GB. Therefore, TT Score chose EfficientTAM S as the default. If a human gives a single tap, all three models converge to 7–8px (TT Score Model Anatomy Part 3 Section 5 Addendum, repository docs/tt/PERFORMANCE.md 2.12).
• The paper reported that EfficientTAM-S/2 scored slightly lower than SAM 2 on SA-V (74.0 vs 74.7), but in our table tennis table detection, EfficientTAM was more accurate. Since the comparison target (SAM 2.1 tiny) and tasks differ, we do not read this as contradicting the paper. The question from "Try it yourself" above, namely how smooth the actual memory map is, has not been measured yet (unmeasured).
• "Segmentation inside the phone" targeted by EfficientTAM is eventually a way to perform table tennis table detection without a server. Since it only needs to be done once at the start of the match, 450ms per frame is also usable.

The next post is about SAM 2, which calls objects by textinstead of clicks.

TT Word. Aggregating memories into 2×2 is fine only when neighboring cells resemble each other. When we aggregated arbitrary memories, the answer changed by as much as 75%.

References

• Ravi, N. et al. (2025). SAM 2: Segment Anything in Images and Videos. ICLR 2025. https://arxiv.org/abs/2408.00714, repository https://github.com/facebookresearch/sam2 (Apache-2.0)
• Xiong, Y. et al. (2024). Efficient Track Anything. arXiv. https://arxiv.org/abs/2411.18933, repository https://github.com/yformer/EfficientTAM (Apache-2.0)
• Ryali, C. et al. (2023). Hiera: A Hierarchical Vision Transformer without the Bells-and-Whistles. ICML 2023. https://arxiv.org/abs/2306.00989
• Our Measurement: TT Score Model Anatomy Part 3, repository docs/tt/PERFORMANCE.md 2.4·2.12
• Example code and output: in this folder, code/10_memory_pool.py, code/10_memory_pool.out.txt