Video-Language Models and Streaming: How They Are Trained and Why They Are Weak in Sports Benchmarks

Status: Draft (2026-10-13), unmeasured. The figures in the paper and technical report were verified directly from the original sources along with their conditions. The only part directly experimented with is the token-counting calculation (code/12_video_tokens.py), and we have never run a Video-LLM on our own table tennis videos (unmeasured). The video input conditions and slow verification layer design of real services (Gemini) are covered in TT Score Model Dissection Part 8. This part focuses on "Training Methods and Benchmarks". Series "Reading Vision Models 2", Part 12 (Final). Part 1's CLIPimage encoder appears here again.

Table of Contents

• 1. Intuition: Attaching Eyes to a Language Model
• 2. How They Are Trained
  • LLaVA: Two Stages
  • Expanding to Video
  • Qwen2.5-VL: Reducing Tokens and Injecting Visuals
  • Streaming: Learning Even When to Speak
• 3. Trying It Out: Token Value of a 10-Second Rally
• 4. What Benchmarks Show
  • TennisTV: Understanding Tennis Rallies
  • F3Set: Dropped from Comparison Entirely
  • SportR: Rule Inference via Reinforcement Learning
  • SCBench: Table Tennis Commentary
• 5. Why They Are Weak: Three Reasons from the Training Method
• 6. Limitations (Regarding the Claims of This Article)
• 7. Connection to Our Project
• References

1. Intuition: Attaching Eyes to a Language Model

Video-language models typically connect two existing components: an image-viewing encoder(the CLIP family from Part 1) and a text-handling language model. A small connection layeris placed between them to convert vectors produced by the image encoder into vectors that the language model can read like words ("image tokens"). For videos, a few frames are extracted, converted into image tokens respectively, and concatenated.

The training loss is the same as for language models: next-token prediction. A question like "How many times did the player hit the ball in this video?" and image tokens are fed in, and the model is trained to predict the words of the correct answer sentence one by one. Within this simple framework, what is shown, how densely it is shown, and what sentences are provided as correct answers determine the model's capabilities.

2. How They Are Trained

LLaVA: Two Stages

LLaVA (NeurIPS 2023) established the baseline sequence for this family (paper).

1. Connection layer training only. Using 595,000 image-caption pairs, the image encoder (CLIP ViT-L/14) and language model are frozen, and only a single linear connection layer is trained. 1 epoch, about 4 hours. This is the stage where image tokens are made to "fit" into the word space of the language model.
2. Instruction-following training. Using 158,000 question-answer format data, the connection layer and language model are trained together. 3 epochs, about 10 hours. The loss is applied only to the words in the answer portion .

Both stages were performed using 8 datacenter-grade GPUs. An interesting point is the origin of the data for the second stage. It was not written by humans; instead, a text-only model unable to see images (GPT-4) was given captions and object bounding box coordinates in text form and asked to write conversations, detailed descriptions, and reasoning QA. This means the answer sentences came from "text about the image" rather than the image itself.

Expanding to Video

Video-LLaVA (2023) uniformly extracts 8 framesfrom a video and feeds them in at 224px. It used 558,000 image-text pairs and 702,000 video-text pairs in stage 1, and 665,000 image instructions and 100,000 video instructions in stage 2 (paper). Several seconds of video are summarized into 8 frames.

Qwen2.5-VL: Reducing Tokens and Injecting Visuals

To feed longer videos, tokens must be conserved. Qwen2.5-VL (2025 Technical Report) does this as follows.

• After dividing the image into 14×14 pixel patches, neighboring 2×2 patches are merged into a single token using a small MLP. A single token handles 28×28 pixels.
• Consecutive two frames are processed as a single bundle.
• During training, the framerate is varied upon input (dynamic FPS), and the time axis of positional encoding is matched to actual seconds. Since the interval of token numbers reflects the actual time interval, the model can answer "at what second".
• During evaluation, a single video was segmented into a maximum of 768 frames and 24,576 tokens.

In the task of finding event segments within a video (Charades-STA), the 72B model achieved an mIoU of 50.9 (report). It handles "second intervals" to some extent, but the unit is in seconds.

Streaming: Learning Even When to Speak

Real-time commentary differs from conversation where one answers after a question finishes. Videos stream continuously, and the model must determine for itself when to speak .

• VideoLLM-online (CVPR 2024): In addition to the language model loss, cross-entropy loss (streaming EOS) is applied to predict "not speaking right now" for each frame. It was trained using timestamped descriptions of egocentric cooking and task videos (Ego4D) and conversations generated by language models. With 2fps input, the paper experiments reduced it to 1 token per frame (the open-source model uses 10), processing a 5-minute video at over 10 frames per second on a single datacenter-grade GPU (paper).
• LiveCC (CVPR 2025): Automatic subtitlesfrom YouTube videos were interleaved with frames (2fps) on a 1-second basis, training the model to predict the words spoken in that 1 second. The timing of the commentator's speech itself becomes the ground truth. Pre-training used 5 million clips, and fine-tuning used 526,000 subtitles recreated via speech recognition (paper).
• StreamingVLM (ICLR 2026): To handle infinitely continuous video, it retains only the first 512 tokens (attention "anchors"), the 512 most recent text tokens, and the visual tokens from the last 16 seconds, discarding the rest. Training is performed by cutting 24-second segments with 12-second overlaps, and for seconds without commentary, "..." is set as the target. Training data consisted of over 4,000 hours across 2,449 games of basketball, soccer, ice hockey, baseball, and American football. There is no table tennis. It ran at 8fps on a single datacenter-grade GPU (paper).

In all three cases, the target answer is human speech (descriptions, subtitles, commentary) . Even if a commentator says "Great forehand!", they do not say "The bounce on the receiver's side was at the 41st frame." The temporal resolution that the model can learn is hard-pressed to exceed the temporal resolution of the answer sentences.

3. Trying It Out: Token Value of a 10-Second Rally

code/12_video_tokens.py We calculated the number of visual tokens required for a 10-second rally using Qwen2.5-VL's approach (28×28 pixels per token, two frames per bundle) (code/12_video_tokens.out.txt). The evaluation upper limit is 24,576 tokens per video.

We then compared the size of the ball at those resolutions against a single token cell. The median ball diameter of 11 pixels in a 1080p side-view video (measured in TT Score Model Dissection Part 2) was scaled down proportionally.

448×252: 공 지름 2.6px = 토큰 한 칸 폭의 9%
896×504: 공 지름 5.1px = 토큰 한 칸 폭의 18%
1792×1008: 공 지름 10.3px = 토큰 한 칸 폭의 37%

A trade-off is visible. To input all frames at 60fps, even if the resolution is lowered to 448×252, it exceeds the upper limit, and at that point, the ball is 9% of a single token. If you increase the resolution so the ball is visible, you can only fit 1–2fps. Since a single bounce contact occurs within one or two frames (17–33ms at 60fps), there is no room in the token budget to simultaneously capture "dense time" and "small objects." For reference, the open-source VideoLLM-online model drastically reduced this to 10 tokens per frame (3×3 grid + 1). That is 200 tokens for 10 seconds at 2fps, but it views the entire screen as merely a 3×3 grid.

4. What Benchmarks Show

TennisTV: Understanding Tennis Rallies

TennisTV constructed 2,527 questions across 8 tasks using 1,298 tennis videos from F3Set (currently based on v5, paper).

Even the best model performs at around half the level of humans and is only 17 points higher than random guessing. Counting hit counts in a rallywas 92.5 for humans and 39.8 for the best model. This aligns with our observation in Part 1 that CLIP is weak at "counting".

Even more interesting is the experiment increasing the number of frames(Qwen2.5-VL-7B, paper).

Even when quadrupling the frames, the overall score remained nearly unchanged, and rally-level tasks actually dropped. Showing more frames does not mean the model knows how to use them. This can be interpreted as stemming from the fact that it rarely saw such correct answers (QA about dense time) during training (estimation).

F3Set: Dropped from Comparison Entirely

F3Set (ICLR 2025), a benchmark for fast and dense fine-grained events, conducted a preliminary test showing GPT-4V 12 consecutive frames of a tennis rally. While it recognized sports, scoreboards, and tournaments, it failed to capture fine-grained events and the order between frames(such as hit direction), leading to its exclusion from the main experiment comparisons (paper appendix). This means it was not at a level to be placed in the same table as the frame-level models from Part 10.

SportR: Rule Inference via Reinforcement Learning

SportR (2025) asks questions on rule and foul inference using 4,789 images across 5 sports and 2,052 videos across 4 sports. The results split into two directions (paper).

• Video foul classificationachieved only 34.39% even for the best model (GPT-5), and the task of pointing out locations in images (IoU) was 4.61 for the baseline model, and 9.94 even after the training described below.
• On the training side, there are also hopeful results. When Qwen2.5-VL-7B was supervised-fine-tuned briefly using only images , followed by reinforcement learning (GRPO) using correctness rewards (graded by another language model or bounding box overlap), its video foul detection accuracy jumped from 25.49% to 59.52% without ever training on videos, surpassing GPT-5 (59.17%).

While inferenceregarding rules can be improved through training, the material for that inference— dense perception(where and when the ball hit)—remains weak.

SCBench: Table Tennis Commentary

SCBench (2024) contains 719 table tennis clips (averaging 9.96 seconds). The commentary score for the best table tennis model (InternVL-Chat-2, 40B parameters) was 5.92 out of 10. Scoring was performed by another language model across six dimensions (paper).

5. Why They Are Weak: Three Reasons from the Training Methodology

1. The ground truth consists of words. The ground truth in training data consists of captions, subtitles, commentaries, and Q&A written by language models. Humans do not speak in bounce frame numbers. Temporal resolution that is absent from the ground truth is difficult to learn.
2. Token budgets force a trade-off between time and space. As calculated above, including all frames makes the ball smaller than a single token, whereas making the ball visible results in sparsely sampled frames.
3. The "eyes" are weak at "counting" and "fine-grained details." Many models use frozen CLIP-family image encoders. We saw in Part 1 that these encoders are weak at counting and fine-grained distinctions.

6. Limitations (Regarding the Claims of This Part)

• Most benchmarks consist of broadcast footage. It is uncertain whether the same results would hold for practice videos recorded with a fixed side-angle camera.
• Models change rapidly. The figures above reflect models at the time of the 2025–2026 papers. Even TennisTV changed its metrics from v1 to v5, so the specific version must be cited when referencing them.
• We have not run any Video-LLMs on table tennis videos ourselves (unmeasured). The conclusions in this part rely on papers and calculations.

7. Connection to Our Project

• Not used for adjudication. Video-LLMs are not used in TT Score's adjudication pipeline (recognition → events → referee). For rallies flagged as "Review Required," we plan to send the clip after the rally ends to ask again for the winner and context, and simply display a slow review layerif it differs from the verdict (TT Score Part 8). We have never actually called them in practice.
• Streaming models are useful for commentary. Models that learn commentary, such as StreamingVLM and LiveCC, are suited for "highlight descriptions" or "rally summaries" rather than "score adjudication." Moreover, we could not find any public models trained on table tennis data.
• Rule reasoning is a different story. As seen in the results of SportR, reasoning about rules can be improved through training. Converting the event sequence generated by our engine (bounce, hit, net) into text and asking "Is this sequence permissible under the rules?" may align better with these models' strengths than directly watching the video (estimated, unmeasured).

A word from TT. Video-LLMs learn from commentators' words. But commentators do not say, "Bounce on the 41st frame!" That is why they speak well, but do not really know when the ball landed. It seems right to leave the adjudication to the ball-counting engine and the descriptions to the articulate model.

References

• Liu et al., Visual Instruction Tuning(LLaVA), NeurIPS 2023. https://arxiv.org/abs/2304.08485
• Lin et al., Video-LLaVA: Learning United Visual Representation by Alignment Before Projection, arXiv 2023. https://arxiv.org/abs/2311.10122
• Bai et al., Qwen2.5-VL Technical Report, arXiv 2025. https://arxiv.org/abs/2502.13923
• Chen et al., VideoLLM-online: Online Video Large Language Model for Streaming Video, CVPR 2024. https://arxiv.org/abs/2406.11816
• Chen et al., LiveCC: Learning Video LLM with Streaming Speech Transcription at Scale, CVPR 2025. https://arxiv.org/abs/2504.16030
• Xu et al., StreamingVLM: Real-Time Understanding for Infinite Video Streams, ICLR 2026. https://arxiv.org/abs/2510.09608
• Bao et al., TennisTV: Do Multimodal Large Language Models Understand Tennis Rallies?, arXiv 2025(v5). https://arxiv.org/abs/2509.15602
• Liu et al., F³Set: Towards Analyzing Fast, Frequent, and Fine-grained Events from Videos, ICLR 2025. https://arxiv.org/abs/2504.08222
• Xia et al., SportR: A Benchmark for Multimodal Large Language Model Reasoning in Sports, arXiv 2025. https://arxiv.org/abs/2511.06499
• Ge et al., SCBench: A Sports Commentary Benchmark for Video LLMs, arXiv 2024. https://arxiv.org/abs/2412.17637
• Example code and output: code/12_video_tokens.py, code/12_video_tokens.out.txt