Video Models: 3D Convolutions, Two-Stream (SlowFast), Attention (TimeSformer) — and Masked Video Modeling (VideoMAE)

Status: Draft (2026-10-13). Paper metrics were verified against the original sources along with their conditions. Hands-on experiments use a numpy example on a notebook CPU (code/06_video_cost.py, under 1 second). These video models have not been tested on table tennis footage (unmeasured). Series "Reading Vision Models 2", Part 6. Previous part: Optical Flow and RAFT. For 2D convolutions and ViT, see Reading Vision Models 1.

Table of Contents

• 1. Intuition: What a Single Image Misses
• 2. 3D Convolutions: C3D and I3D
  • C3D: Three Temporal Frames
  • I3D: Starting by Inflating an Image Model
  • Cheap Alternative: Temporal Shift Module (TSM)
• 3. Two-Stream: Two-Stream and SlowFast
• 4. Attention: TimeSformer
• 5. Hands-on Experiments
• 6. How They Are Trained: Self-Supervised Masked Video Modeling
  • VideoMAE: 90% Masking with Tubes
  • V-JEPA: Predicting Features Instead of Pixels
• 7. Paper Metrics
• 8. Limitations
• 9. Connection to Our Project
• References

1. Intuition: What a Single Image Misses

A single still image can tell you that there is a "player holding a racket," but it cannot tell you whether the player is in the middle of hitting, has just finished hitting, or is preparing to hit. "Opening a door" and "closing a door" look identical in a single photo. Video models aim to learn this difference—namely, information contained in temporal order.

There are broadly three approaches.

1. 3D Convolution: Extends filters from height and width to also cover the temporal dimension.
2. Two-Stream: Separates the pathway that looks at appearance from the pathway that looks at motion.
3. Attention: Tokens cross space and time to reference one another.

Regardless of the approach, training is mostly supervised. Classification is learned using video-level labels like "This clip is 'playing table tennis'." A representative dataset is Kinetics (approx. 240k training videos, 400 action classes). Only VideoMAE in the final section uses unsupervised pre-training.

2. 3D Convolutions: C3D and I3D

C3D: Three Temporal Frames

C3D (2015) extended convolution filters from 3×3 to 3×3×3. Taking 16-frame × 112×112 clips and testing temporal depths of 1, 3, 5, and 7, it found 3 to be the best (the paper notes that this was compared over a narrow range). It was trained from scratch on 1.1 million sports videos (Sports-1M, 487 classes).

I3D: Starting by Inflating an Image Model

The weakness of 3D convolutions is that they must learn everything from scratch. While images have well-trained weights from ImageNet, videos lacked a comparable starting point. I3D (2017) introduced a simple trick to convert 2D weights into 3D.

Replicate 2D filters N times along the temporal dimension and divide by N.

3D filters constructed this way produce the exact same answers as the original 2D filters on a "boring video" where frames never change. Training starts by inheriting what the image model already knows, while motion is learned along the way.

Cheap Alternative: Temporal Shift Module (TSM)

3D convolutions triple parameters and compute costs. TSM (2019) keeps the 2D convolutional network intact, simply shifting ⅛ of the feature channels forward by one frame and ⅛ backward by one frame. This introduces information from neighboring frames without adding a single multiplication. By restricting shifts backward only (using past frames only), it can be used for real-time streaming (paper). GSM, used by E2E-Spot in Part 10, is a variant that learns "what to shift and by how much."

3. Two-Stream: Two-Stream and SlowFast

Two-Stream (2014) trains a pathway looking at a single image (spatial stream) and a pathway looking at a stack of 10 optical flow frames from Part 5 (temporal stream) separately, averaging their scores at the end. The temporal stream alone was more accurate than the spatial stream (UCF-101 83.7% vs. 73.0%, paper). This indicates how crucial motion is for action recognition, but it is slow because optical flow must be pre-computed.

SlowFast (2019) builds two streams without optical flow.

• Slow pathway: Processes frames sparsely (e.g., 1 out of every 16 frames) with many channels to capture what is present.
• Fast pathway: Processes frames densely at 8x frequency (α=8) with ⅛ of the channels (β=1/8) to capture how things move. Computation is about 20% of the total.
• Features from the fast pathway are fed into the slow pathway at multiple layers (lateral connections).

The authors note this was inspired by the primate visual system, where shape-processing and motion-processing cells are separate. Above all, it was trained from scratch using only Kinetics, without ImageNet pre-training . Even with pre-training, the difference was within ±0.3%p (paper).

4. Attention: TimeSformer

When applying ViT to video, tokens multiply by "number of frames × patches per frame." For 8 frames × 196 patches, that is 1,568 tokens. Joint attention, where every token attends to every token, scales quadratically with the number of tokens.

TimeSformer (2021) divides attention into two steps within a single block (divided attention).

1. Temporal attention: Each patch attends only to patches at the same spatial locationin other frames.
2. Spatial attention: Using that result, each patch attends only to other patches in the same frame.

Dividing it this way dramatically cuts computation while slightly improving accuracy (table below). It starts from an ImageNet pre-trained ViT and fine-tunes on video labels.

5. Hands-on Experiments

code/06_video_cost.py output fromcode/06_video_cost.out.txt).

I3D Inflation. On a "boring video" where a single 12×12 image is repeated 5 times, we applied a 3×3×3 filter created by replicating a 3×3 filter 3 times and dividing by 3.

멈춘 영상에서 3D 출력과 2D 출력의 최대 차이: 1.33e-15
움직이는 영상에서는 차이가 생긴다(평균 절댓값 1.644) — 이 차이를 학습이 키워 움직임 필터가 된다

A difference of 10⁻¹⁵ simply reflects floating-point rounding error. They are identical.

Attention Computation Cost. We counted the number of token pair comparisons in a single layer (16×16 patches, 224 resolution means 196 patches per frame).

The gap widens as frame count increases. This factorization is why TimeSformer-L can process 96 frames.

6. How They Are Trained: Self-Supervised Masked Video Modeling

VideoMAE: 90% Masking with Tubes

VideoMAE (2022) adapts Part 3's MAE for video. There are two main differences.

• Higher masking ratio. While images use 75%, videos use 90%(95% was also fine). Because adjacent frames are nearly identical, videos contain much more redundancy than images.
• Tube masking. When a spatial location is masked, the same location is masked across all frames. A 2-frame × 16×16 pixel chunk (cube) is treated as a single token.

We measured why tube masking is necessary. In a 16-frame × 196-patch setup, we looked at the ratio of masked patches where the exact same locationis visible in the immediately preceding or succeeding frame.

가림 75% · 프레임마다 따로        : 0.415
가림 90% · 프레임마다 따로        : 0.180
가림 95% · 프레임마다 따로        : 0.092
가림 75–95% · 튜브(모든 프레임 같은 자리): 0.000
옆 프레임 같은 자리를 그대로 베낀 오차 0.0120  (아무것도 모르고 0 으로 채우면 1.0009)

If you mask frames independently, even with 90% masking, 18% of the patches can be guessed almost perfectly (error 0.012, vs. 1.0 for random guesses) simply by copying from the same spot in the neighboring frame. The model learns the lazy shortcut of "copying" rather than learning motion or structure. Tube masking blocks this shortcut entirely.

The loss function is identical to MAE: predicting the (normalized) pixels of the masked cubes using mean squared error. The encoder processes only the visible ~10% of tokens.

V-JEPA: Predicting Features Instead of Pixels

V-JEPA (2024) replaces pixel prediction for masked regions with features produced for those regions by a slowly lagging teacher encoder (Part 2's EMA teacher), optimized via L1 loss. The idea is to avoid wasting capacity on trivial pixel-level details (connecting to the limitations discussed in Part 3). It uses no pre-trained image models, text, negative examples, or pixel reconstruction (paper). features, optimized via L1 loss. The idea is to avoid wasting capacity on trivial pixel-level details (connecting to the limitations discussed in Part 3). It uses no pre-trained image models, text, negative examples, or pixel reconstruction (paper).

7. Paper Metrics

Top-1 accuracy on Kinetics-400. "Views" refers to (number of clips sampled per video during evaluation) × (spatial crops per clip).

A comparison of attention designs in TimeSformer (with ImageNet-21K pre-training) is shown below (paper).

Note that on Kinetics, even when ignoring time, accuracy is 76.9% . Many actions in Kinetics can be recognized just from the scene (swimming pool, ping-pong table). In contrast, Something-Something requires understanding motion itself(e.g., "pushing something from left to right"), so ignoring time causes accuracy to collapse to 36.6%. Table tennis classification definitely leans toward the latter.

Training Costalso varies. Under ImageNet-1K pre-training conditions, training on Kinetics took 416 GPU hours for TimeSformer, 3,840 hours for SlowFast R50, and 1,440 hours for I3D (on datacenter-class GPUs, TimeSformer paper).

VideoMAE Data Efficiency. Results when pre-training the same ViT-B using only the training videos of each dataset (paper).

The difference is most pronounced on small datasets with only a few thousand videos (the UCF101 and HMDB51 experiments were pre-trained with 75% masking). Also, the version pre-trained on 42k Something-Something videos (68.7) performed better on Something-Something than the version pre-trained on 240k Kinetics videos (68.5). The paper cites this as evidence that "for pre-training data, the degree of similarity to the target domain is more important than sheer volume."

8. Limitations

• Classification does not tell you "when." The models in this post assign a single label to the entire clip. While they know that "a hit occurs within these 2 seconds," they do not know "which specific frame." Frame-level timing is covered in Part 10: Precise Event Detection.
• Bidirectionality is not real-time. 3D convolutions and attention typically look at both the past and the future of a clip. Since real-time applications have no future, you must either make it causal—like TSM's unidirectional version—or accept a short delay (look-ahead).
• Small objects disappear in the resolution. In frames resized to 224px, a table tennis ball might occupy less than a single pixel.

9. Connection to Our Project

• Our ball detector is a very simple form of approach ①. BlurBall and WASB stack 3 consecutive frames into channels (9 channels) and feed them into a 2D convolutional network (covered in TT Score Model Anatomy, Part 2). This is "early fusion," where time is mixed once at the first layer. TOTNet, which handles masking in Part 9, builds on this by adding 3D convolutions.
• Our learning-based boundary classifier applies temporal convolution to trajectories rather than pixels. Because it uses 1D convolution on a time series of 9 features like ball coordinates and velocity, the computational cost is very low (Part 4, research ongoing as of research lead status on 2026-10-13).
• Video-level action recognition is a candidate for coaching. Stroke classification—such as "forehand drive or block"—is a problem suited for this type of model. Current coaching is posture (joint)-based, and video-based models have not yet been tested (unmeasured).

A word from TT. "Before the hit" and "after the hit" look identical in a single photo. However, if you mask frames individually and ask the model to guess, it learns the trick of copying adjacent frames instead. Tube masking is a mechanism to prevent that shortcut.

References

• Tran et al., Learning Spatiotemporal Features with 3D Convolutional Networks(C3D), ICCV 2015. https://arxiv.org/abs/1412.0767
• Simonyan & Zisserman, Two-Stream Convolutional Networks for Action Recognition in Videos, NeurIPS 2014. https://arxiv.org/abs/1406.2199
• Carreira & Zisserman, Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset(I3D), CVPR 2017. https://arxiv.org/abs/1705.07750
• Lin, Gan, Han, TSM: Temporal Shift Module for Efficient Video Understanding, ICCV 2019. https://arxiv.org/abs/1811.08383
• Sudhakaran et al., Gate-Shift Networks for Video Action Recognition(GSM), CVPR 2020. https://arxiv.org/abs/1912.00381
• Feichtenhofer et al., SlowFast Networks for Video Recognition, ICCV 2019. https://arxiv.org/abs/1812.03982
• Bertasius, Wang, Torresani, Is Space-Time Attention All You Need for Video Understanding?(TimeSformer), ICML 2021. https://arxiv.org/abs/2102.05095
• Tong et al., VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training, NeurIPS 2022. https://arxiv.org/abs/2203.12602
• Bardes et al., Revisiting Feature Prediction for Learning Visual Representations from Video(V-JEPA), TMLR 2024. https://arxiv.org/abs/2404.08471
• Example code and output: code/06_video_cost.py, code/06_video_cost.out.txt