Finding the Table Tennis Table: SAM 2.1, EfficientTAM, SAM 3, and Homography from Four Corners

Status: Draft (2026-10-13), actual measurements reflected. The accuracy and speed of SAM 2.1 tiny were measured on 2026-10-11–12 using a server GPU (shared with other tasks at the time) and a temporary CPU pod. EfficientTAM and SAM 3 were not run in our environment (unmeasured); their figures from papers and official repositories are cited along with their conditions. Automatic table tennis table detection is currently under development in another task and will be added once results are available. Authoring rule: Actual measured data is the baseline, and unmeasured items are marked as "Unmeasured".

Table of Contents

• 1. Why Four Corners Are Needed
• 2. Prompt Segmentation: A Single Click Becomes a Question
  • How SAM Thinks
  • SAM 2: Connecting Videos with Memory
  • EfficientTAM: Slimming Memory for Mobile Phones
  • SAM 3: Calling by Text
• 3. From Mask to Four Corners
  • Two Failed Approaches
  • Current Approach: Refitting Straight Lines to Each Edge
  • Isolating the Top Surface
• 4. From Four Corners to the Table Plane: Homography
  • Intuition
  • Our Code
  • Two Pitfalls
  • Why Go All the Way to the Plane
• 5. Measurements
  • Corners of SAM 2.1 tiny (2026-10-11)
  • Speed and Memory
  • Impact on Scoring
  • What Was Excluded
  • Outside of OpenTTGames
• 6. Conclusion and Next Steps
• References

1. Why Four Corners Are Needed

Ball coordinates alone are not enough to score points. To determine whether "the ball touched the table," "which half it touched," or "whether it went past the edge of the table," we need to know where the table is located on the screen. A table tennis table is a rectangle with regulation dimensions: 2.74m long, 1.525m wide, and a net height of 15.25cm. Therefore, if we find the four cornerson the screen, the rest can be calculated using geometry.

Fortunately, TT Score assumes that the camera is fixed on a tripod. Since the table does not move, it only needs to be found once at the start of the match . Unlike ball detection, which must run every frame, spending several hundred milliseconds here is acceptable. This difference changes the model selection.

Currently, the live screen requires a human to manually tap the four corners. This is the most reliable method and requires zero computation. If the user taps the two halves of the table once more each, the engine can select which side is the endline even with an angled camera (Section 5 below). The models in this post are "ways to reduce or eliminate manual taps."

2. Prompt Segmentation: A Single Click Becomes a Question

How SAM Thinks

The Segment Anything (SAM) family does not learn "what to color" during training. Instead, it receives promptsprovided by a human (point clicks, boxes, or rough masks) and colors the object at that location. The architecture consists of three parts:

1. Image Encoder: The heavy part. It runs a single image once to generate a feature map.
2. Prompt Encoder: Converts click coordinates or boxes into small vectors.
3. Mask Decoder: The lightweight part. It receives the feature map and prompts to output a mask.

Because the heavy encoder runs only once while the lightweight decoder runs repeatedly on every click, it is well-suited for interactive screens where a user modifies clicks iteratively. In TT Score, the human's taps pointing to the table directly become the prompt.

SAM 2: Connecting Videos with Memory

SAM 2 (arXiv 2408.00714) extends this concept to video. An object pointed to in the first frame must be tracked across subsequent frames. To achieve this, it uses memory.

• Memory Encoder: Downsamples the predicted mask for each frame, adds it to the image features of that frame, and blends them with lightweight convolutions to create a memory of "what the object looks like in this frame."
• Memory Bank: Maintains separate queues for recent frame memories (default of 6) and memories from frames where human prompts were provided. It also stores small vectors (object pointers) extracted from the mask decoder's output tokens.
• Memory Attention: Features of the new frame first attend to themselves (self-attention), and then attend to the memories in the memory bank and object pointers (cross-attention). Features passing through 4 such blocks go to the mask decoder.

Intuitively, this means "find the object in the current frame by referencing how it looked in the past few frames." Even if a player momentarily blocks a table corner, the memory persists so the mask does not break.

The video API provided by the official repository consists of three steps.

# facebookresearch/sam2 README.md(Apache-2.0)에서
predictor = build_sam2_video_predictor(model_cfg, checkpoint)
with torch.inference_mode(), torch.autocast("cuda", dtype=torch.bfloat16):
    state = predictor.init_state(<your_video>)
    frame_idx, object_ids, masks = predictor.add_new_points_or_box(state, <your_prompts>)
    for frame_idx, object_ids, masks in predictor.propagate_in_video(state):
        ...

Here are the metrics by size for SAM 2.1 (September 2024) (official README, single datacenter-class GPU, all components compiled).

Both code and checkpoints are licensed under Apache-2.0. We chose the smallest tiny model because a table tennis table is a large, simple object where the benefits of a larger model would be small, and because resources must be shared with tasks like ball detection.

EfficientTAM: Slimming Memory for Mobile Phones

EfficientTAM (arXiv 2411.18933, code and checkpoints under Apache-2.0) introduces two optimizations to make SAM 2 lightweight:

1. Flat ViT encoder. Instead of SAM 2's hierarchical encoder (Hiera), it uses a single-resolution small ViT (Tiny/Small).
2. Coarse memory cross-attention. Spatial tokens in memory are averaged over 2×2 windows to reduce them to a quarter before attention. The paper explains that due to "local smoothness"—where memories of neighboring pixels are similar—these reduced tokens effectively mimic the original attention (relative error of 0.03 compared to full attention). Object pointers are not reduced.

According to the paper's table (1024×1024 input), EfficientTAM-S/2 achieves 74.0 J&F on SA-V with 34M parameters, running at 109.4 FPS on a datacenter GPU and 450ms per frame on a mobile phone (iPhone 15 Pro Max). The compared SAM 2 model has 81M parameters, 74.7 J&F, and 43.8 FPS (from the paper). The precision and conversion methods for the mobile measurements are not detailed in the paper, and no direct comparison with SAM 2.1 is provided.

For us, this represents "a path to finding table tennis tables inside mobile phones someday." Since it only needs to run once at the start of a match, 450ms per frame is quite usable. We have not yet run it in our environment (unmeasured).

SAM 3: Calling by Text

SAM 3 (arXiv 2511.16719, November 2025) introduces promptable concept segmentation. When provided with a short text query like "table tennis table" or an example image, it finds and colors all objects corresponding to that concept and tracks them across video. It is attractive because it can find the table tennis table without clicks.

However, we excluded it this time for several reasons:

• License. It is not Apache, but uses a separate "SAM License." Conditions include attaching the same agreement upon redistribution, prohibitions on export control and military use, bans on reverse engineering, and termination upon litigation. Commercial use is not explicitly addressed.
• Access Approval. To download the checkpoints, you must apply for access on Hugging Face.
• Size. It has 848M parameters (official README), which is over 20 times larger than tiny. The paper reports 30ms for a single image with over 100 objects on a datacenter GPU.
• Necessity. There is only one table on the screen, and a human can simply point to it once. The advantage of "finding all concepts" is minimal here.

3. From Mask to Four Corners

What a segmentation model outputs is a mask (a set of colored pixels). What we need are four vertices. This conversion turned out to be trickier than expected.

Two Failed Approaches

Initially, we simplified the mask contour into a polygon to pick vertices (approxPolyDP). When the net divided the top surface into two halves, it treated them as half-tables; when a player blocked a corner, it output an incorrect inner point as a corner.

Next, we tried matching edges using the rule "top of each column in the mask = distant edge, bottom = near edge." This worked well for side views, but in videos where the table was viewed at an angle (such as angled camera footage from the TT3D lab), a single vertex was off by 370–754px . This happened because the end edge was misread as the distant edge, causing the side accuracy to drop from 0.99 to 0.80.

Current Approach: Refitting Straight Lines to Each Edge

ttscore/calib.py works as follows:

1. Collect exterior boundary points of the mask.
2. Roughly determine the four edges using a bounding rectangle around the convex hull.
3. For each edge, refit a straight line using only boundary points near that edge . To remain robust even when nearly half of the points are corrupted by net posts or player arms, we first determine the line slope using repeated medians (Siegel regression), and then iteratively apply least squares using only points within 3px of that line.
4. The intersectionof two adjacent edges serves as a vertex.
5. Instead of using just a single frame, vertices from multiple frames (5 frames in our measurements) are combined using the median.

# ttscore/calib.py _fit() 에서 줄임 — v = a + b·u 직선, 순수 파이썬
slopes = [_median([(vs[j] - vs[i]) / (us[j] - us[i]) for j in idx if us[j] != us[i]]) for i in idx]
b = _median(slopes)                       # 반복 중앙값: 점의 절반이 어긋나도 기울기가 버틴다
a = _median([vs[i] - b * us[i] for i in idx])
for _ in range(it):                        # 그 직선에서 tol(3px) 안의 점만으로 최소제곱을 되풀이
    keep = [i for i in range(n) if abs(vs[i] - (a + b * us[i])) <= tol]
    ...

The advantage of this method is that occluded vertices are determined by two visible edges. Even if a player is covering a corner, as long as the remaining parts of the two edges forming that corner are visible, the intersection ends up in the correct place. With the same SAM mask, the maximum vertex error on angled footage dropped from 370–754px down to 25–28px , and the side accuracy rose to 0.999.

Since CI lacks numpy, this module is implemented in pure Python. Masks are received as colored span intervals per row {y: [(x0, x1), ...]} .

Isolating the Top Surface

SAM sometimes colors side panels and legs in addition to the table top, or conversely colors only one half due to obstruction by the net. Therefore, we provide two taps (one for each of the two halves divided by the net) and retain only pixels with colors similar to the tapped locations to separate the top surface from the side panels. This step was vulnerable to shadows. In two side-view lab videos, color thresholds cut off the shadowed edges, causing vertices to be off by 87–97px. Such quadrilaterals are caught by a validation check (quad_ok) which rejects them as non-rectangles, causing the system to fall back to previous rules and preserve a side accuracy of 0.99.

Because the recorded video analysis worker does not yet have SAM 2.1 loaded, it currently only performs the color grading step of keying similar colors at the tab location (batch/calib.py). The corner accuracy of this method was not measured (unmeasured), so we document the method in the results and recommend "direct four-corner pointing" for the screen.

4. From Four Corners to Table Plane: Homography

Intuition

When a flat surface is captured by a camera, a rectangle appears as a trapezoid. This "plane-to-plane" perspective transformation is called a homography. It is expressed as a single 3×3 matrix, and fixing the last element to 1 leaves 8 unknowns. Since a point correspondence provides two equations (x, y), four points are sufficient to solve it exactly. You simply map the four corners of the screen to the four vertices of the table: (0, 0) … (2.74, 1.525).

Our Code

Using OpenCV's getPerspectiveTransform(four points) or findHomography(multiple points, RANSAC) takes only a single line, but the engine must run without numpy or OpenCV, so it solves the 8×8 system of linear equations directly.

# ttscore/table.py
def homography(src, dst):
    """네 점 대응 src[i] → dst[i] 의 3×3 변환(h33 = 1)."""
    a, b = [], []
    for (x, y), (u, v) in zip(src, dst):
        a.append([x, y, 1, 0, 0, 0, -u * x, -u * y]); b.append(u)
        a.append([0, 0, 0, x, y, 1, -v * x, -v * y]); b.append(v)
    h = _solve(a, b)                       # 부분 피벗 가우스 소거
    return ((h[0], h[1], h[2]), (h[3], h[4], h[5]), (h[6], h[7], 1.0))

This transformation maps a point on the screen to table plane coordinates (u, v). u represents the longitudinal direction with the center (net) as 0. This results in consistent equations regardless of the viewpoint.

• Side: u < 0 means L end, u > 0 means R end.
• Net: u = 0.
• Beyond the end: |u| > 1.37m.

Two Pitfalls

First, plane projection is only valid at the exact moment the ball touches the table. Mapping a ball in mid-air to the plane yields the "point where the line passing through the ball from the camera intersects the table plane," meaning higher balls are pushed further away. In end cameras, a ball at net height is displaced by over 50cm. Therefore, plane coordinates are used only at the moment of the bounce, while the event rules in Part 5 handle mid-air segments separately. Even at the bounce moment, the ball's center is positioned a radius (20mm) higher than the contact point, so the contact pointunderneath is calculated to determine the side. This single correction raised the winner accuracy for composite oblique cameras from 79.5% to 85.6%.

Second, four points alone do not reveal which edge is the endline. This is because any quadrangle can be mapped to a rectangle in two different ways. The engine determines this in the following order:

1. If data or a human specifies the end edge, use that.
2. If there are two tap points recorded, one on each half of the table, choose the pairing that separates those two points the furthest along u.
3. If the screen size is known, treat the screen center as the principal point and reconstruct the actual aspect ratio by solving for the focal length under the condition that the two vanishing points are orthogonal (rectangle recovery). The pairing closest to 2.74 : 1.525 is correct.
4. If nothing is available, fall back to the old rule (side assumption: the two points with smaller x form one end).

Why Go All the Way to the Plane?

The old engine divided the screen left and right using the x-coordinate of the diagonal intersection ("net x"). While this worked for side views, it presented two problems. The diagonal intersection amplifies the vertical error of the corners by about 8 times (measured: corner 6–15px error → net x 27–36px). Furthermore, in end-side cameras, that vertical line became the center white line across the table width, turning side judgments into a coin toss (composite end camera 0.55, actual broadcast footage 0.60). Converting to plane coordinates enabled consistent rules regardless of viewpoint.

5. Actual Measurements

SAM 2.1 tiny Corners (2026-10-11)

Conditions: 12 actual OpenTTGames matches, top surface only using two tap points + color + fitting straight lines to four edges, 5-frame median. Comparison target: corners obtained from the dataset-provided segmentation mask (1920×1080).

The dataset masks were not always correct. Visually comparing the net baseline with a 3x zoom revealed that in two matches (test_1, game_5), the SAM results were more accurate.

Speed and Memory

Since it only needs to run once at the start of a match, it finishes within a few seconds even on a CPU. The significance of these numbers is that GPU usage is not strictly required because it does not need to run every frame.

Impact on Scores

When only the corners were changed using the same engine on the same day, the winner accuracy across 275 actual match rallies improved from 82.5% (mask corners) to 84.7% (SAM corners) . Bounces near distant edges falling inside the band was the main reason. Later, as the engine changed, accuracy rose up to 93.1% with SAM corners, but since that corner file was no longer available, the 2026-10-13 baseline (91.3%, recorded footage scoring) was remeasured using mask corners. Because the engines also differed, the exact share attributed solely to the corners cannot be isolated. In full-match mode where the scoreboard determines the server, mask corners performed noticeably worse than SAM records, implying that corner quality heavily influences scoreboard flow.

Things Not Included

We tried using SAM 2 for objects other than the ping-pong table, but omitted them as they provided no measured gains.

Outside OpenTTGames

6. Conclusion and Next Steps

• For fixed cameras, the ping-pong table only needs to be found once . Therefore, rather than a massive model, the smallest SAM 2.1 combined with geometrywas sufficient.
• What determined accuracy was not the segmentation model, but the geometry that converts masks into vertices. Refitting straight lines for each edge successfully handled occlusions and oblique viewpoints.
• Once vertices are established, homography maps the screen to the table plane, allowing sides, the net, and ends to be determined consistently via the same rules regardless of viewpoint—though only at the exact moment of a bounce.

Automatic table detection without tabs (detecting and tracking tables in match footage) is currently being developed as a separate task. Those results and actual measurements for EfficientTAM will be appended to this post (in progress).

Titi's Tip. It is fine even if table corners are blocked by a player's arm, because you can simply extend the visible two edges to find their intersection point.

Further Reading on Principles. The original papers and learning principles for the SAM series are covered in Reading Vision Models Parts 1 & 9: Segment Anything, Part 10: SAM 2 and EfficientTAM, Part 11: SAM 3.

References

• SAM 2: https://arxiv.org/abs/2408.00714, https://github.com/facebookresearch/sam2 (Apache-2.0)
• EfficientTAM: https://arxiv.org/abs/2411.18933, https://github.com/yformer/EfficientTAM (Apache-2.0)
• SAM 3: https://arxiv.org/abs/2511.16719, https://github.com/facebookresearch/sam3 (SAM License)
• OpenCV findHomography·getPerspectiveTransform: https://docs.opencv.org/4.x/d9/d0c/group__calib3d.html
• Ping-pong table specifications (length 2.74m, width 1.525m, net height 15.25cm): ITTF Handbook https://www.ittf.com/handbook/
• Our measurements: Repository docs/tt/PERFORMANCE.md 2.4·2.8·2.10