Why Vision-Language Models Fail at Ruling: One Frame per Second, Second-Level Timestamps, and Tennis Benchmarks

Status: Draft (2026-10-13), based on documentation and papers (no internal measurements yet). This article is written based on figures from official documentation (accessed 2026-10-13) and research papers. We did not actually call Vision-Language Models in our experiments. The slow review layer is in the planning stage. Writing rule: Measured data is the default, and unmeasured items are marked as "Unmeasured".

Table of Contents

• 1. An Alluring Idea
• 2. Reason One: One Frame per Second
• 3. Reason Two: What Benchmarks Reveal
• 4. Reason Three: Real-Time and Accountability
• 5. Where to Place It Instead: The Slow Review Layer
• 6. Conclusion
• References

1. An Alluring Idea

Vision-Language Models (VLMs) take in videos and describe what happened in sentences. They also know the rules of table tennis in text. So, why not skip parts 1 through 7—ball tracking, table geometry, event rules, and the referee engine—and simply ask, "Who won this rally?"

To cut to the chase, right now, this is not suitable for determining scores. It is not because the model lacks intelligence, but because the exact moment a point is decided never makes it into the model's input .

2. Reason One: One Frame per Second

The Gemini API video understanding documentation (updated 2026-10-09) states:

• Video input defaults to sampling 1 frame per secondfor processing. fps can be specified separately.
• At low resolution settings, a single frame is 66 tokens, and 1 second of video is about 100 tokens.
• Specific points in time within a video are queried using timestamps in the format MM:SS .
• And the documentation itself warns: "Fast action may lose details due to 1 FPS sampling."

The Live API documentation for real-time conversation is even clearer. Video is sent as individual images at at most one frame per second, and sessions combining audio and video are limited to 2 minutes.

Let us apply this to table tennis.

When capturing one frame per second, the two bounces occurring in a single serve usually happen within the same 1-second window and pass between two samples. "Which side and how many times it bounced" determines the score, but that exact moment is completely missing from the input. The model can only guess based on the surrounding scenes.

fps Could we just increase the rate? The documentation provides a configuration method but specifies no upper limit, while the example actually uses 0.5. Sending at 60fps would increase tokens by 60 times, and we still found no evidence that the model visually pinpoints short contacts between frames. The fact that it handles time in second units is also problematic. Second-level timestamps are far too coarse for judgments where 16.7ms-level sequence order matters.

3. Reason Two: The Level Demonstrated by Benchmarks

TennisTV (arXiv 2509.15602, v5 as of April 2026) is a benchmark measuring how well VLMs perform on racket sport videos. It evaluated 17 VLMs using 1,298 tennis videos, 2,527 questions, and 8 tasks.

The hardest task was counting how many times the ball was hit in a rally, where Gemini 2.5 Pro scored 31.5%. Some models performed worse than random guessing (25%). The paper also notes that commercial models were evaluated at 1fps due to cost constraints. Changing the frame count with an open-source model (Qwen2.5-VL-7B) yielded virtually no change: 38.1 at 1fps, 38.6 for 16 frames, 38.8 for 32 frames, and 38.4 for 64 frames. Giving it more frames does not automatically make it better. The authors conclude that sample density must be tailored to each task.

Since the number of tasks and figures vary slightly across benchmark versions, the specific version must be cited when referencing them. We could not find a VLM evaluation benchmark specifically dealing with table tennis in this survey.

4. Reason Three: Real-time and Accountability

• Latency. The Live API documentation does not provide latency measurements (unverified). The decision budget in Part 1 is 1.5 seconds, and the current engine's median decision latency is 0.16 seconds.
• Sessions. If an audio + video session lasts 2 minutes, sessions must be continuously chained together while watching a single match.
• Explainability. A rule engine can present events one by one, stating, "This bounce occurred on this side at this time, so this player lost." When humans make corrections, they can see exactly where the mistake was. A VLM's text response may sound plausible, but it is difficult to verify which scene it was based on.

5. So Where Does It Fit: The Slow Verification Layer

What VLMs fail at is "sequence within 16.7ms," and what they excel at is "the meaning of the scene as a whole." Coarse events—such as whether a rally has ended, whether a player is making a gesture to count points, or whether they are going to fetch the ball—can be perceived even at one frame per second. Therefore, we plan to place it at the very bottom of the four layers from Part 1, in the Slow Verification Layer.

1. The scoreboard directly displays the rule engine's decisions. It does not wait for the VLM.
2. Only rallies marked as "Verification Required" have clips of a few seconds before and after sent asynchronously to the VLM.
3. If the VLM's answer differs from the referee's, it is merely flagged . A human is the one who changes the score.

The cost is low. Calculated at document pricing rates, re-evaluating a 10-second rally at 1fps takes about 1,300 tokens, which is about $0.0004 on the cheapest tier (calculated based on document rates, unmeasured). However, since this involves sending video to an external service, user consent and data retention policies must be established first. It has not actually been called yet, nor was it called in research experiments (unmeasured).

6. Conclusion

This table will change once vision-language models become capable of handling finer temporal resolution. At that time, it can be re-evaluated with the same question: "Is the exact moment the score is determined included in the input?"

A word from TT: A referee who opens their eyes once a second cannot see where the ball landed. Instead, it might be useful as an assistant that says after a match ends, "Doesn't this scene look strange?"

References

• Gemini API Video Understanding: https://ai.google.dev/gemini-api/docs/video-understanding (Updated 2026-10-09, Accessed 2026-10-13)
• Gemini Live API Guide: https://ai.google.dev/gemini-api/docs/live-guide (Accessed 2026-10-13)
• TennisTV: https://arxiv.org/abs/2509.15602 (v5)
• Actual measurement of bounce intervals: Repository ttscore/referee.py README, events.py 's END_AFTER_BOUNCE_S comment