Optical Flow and RAFT: Reading Motion as Pixels — Comparing every pair once, repeating the same update

Status: Draft (2026-10-13), partially measured. Paper figures were verified by opening the original text and checking conditions. Hands-on trials were done with a numpy example on a laptop CPU (code/05_flow_toy.py, under 1 second). The optical flow measurement (Farneback) for the table tennis video comes from the value in repository docs/tt/PERFORMANCE.md 2.9.7, while RAFT was not run on our video (not measured). Part 5 of the series "Reading Vision Models 2". From here on, it is about "motion". Next part: Video Models.

Table of Contents

• 1. Intuition: One Arrow per Pixel
• 2. Assumptions of Classic Methods and Their Limits
• 3. Where Learning Models Get the Ground Truth
• 4. Figure: RAFT
• 5. How It Is Trained
• 6. Hands-On Trial
• 7. Paper Numbers
• 8. Limitations
• 9. Connection to Our Project
• References

1. Intuition: One Arrow per Pixel

When two frames of a video are overlaid, moving parts are misaligned while stationary parts overlap. Optical flowis a map that attaches an arrow (horizontal movement u, vertical movement v) to every single pixel in the first frame, indicating "where it went in the next frame." For a 1080p frame, there are about 2 million arrows.

Optical flow does not tell us "what" moved, but only "where and how much" it moved. Therefore, it can be used without knowing the objects. In the table tennis video, sections where players move significantly (during rallies) and sections where they linger (between rallies) are noticeably distinguished simply by the magnitude of the optical flow.

2. Assumptions of Classic Methods and Their Limits

Classic methods start from two assumptions.

1. Brightness is constant. Even if a single point moves, its brightness does not change: I₁(x, y) = I₂(x+u, y+v).
2. Motion is small. Thus, the right side can be first-order Taylor expanded: I_x·u + I_y·v + I_t ≈ 0.

There is one equation but two unknowns (u, v). Lucas–Kanade (1981) assumes that "pixels within a small window move together," gathers the equations within the window, and solves them using least squares.

There are three places where this method breaks down:

• Aperture problem. In areas where patterns change in only one direction, such as stripes, motion along the lines leaves no trace on brightness. The equations cannot be solved.
• Large motion. The first-order expansion is only correct when motion is 1–2 pixels. If a fast ball moves tens of pixels in a single frame, it fails. Reducing the image to make large motion small and then scaling it back up, known as a pyramid, is the standard remedy.
• Brightness variation. Shadows, reflections, and motion blur break the first assumption. The blur of a table tennis ball falls squarely into this category.

3. Where Learning Models Get the Ground Truth

It is impossible to manually label correct arrows for every pixel in photographs. Therefore, learning-based optical flow started from synthetic data. FlowNet (2015) rendered and attached 809 types of 3D chair models onto photo backgrounds, creating Flying Chairs(22,872 pairs) moved via affine transformations. Because the motion was artificially created, the ground truth is precisely known (paper). Afterwards, a training pipeline leading to the more realistic FlyingThings3D, Sintel extracted from animated movies, and KITTI for automotive streets became standard.

The loss is typically the End-Point Error (EPE) . It averages the distance (in pixels) between the tips of the predicted arrow and the ground truth arrow.

4. Figure: RAFT

RAFT (Teed & Deng, ECCV 2020) consists of three components.

1. Feature encoder. Passes both frames through the same CNN to extract ⅛-resolution features (256-dimensional). By comparing features instead of brightness, the "brightness is constant" assumption is relaxed.
2. All-pairs correlation volume. Computes the dot product between every position in the first frame and every position in the second frame all at once, storing it as a 4D table (H×W×H×W). It then pools the last two dimensions by 1, 2, 4, and 8 steps to build a 4-level pyramid. Once computed, it is done.
3. Iterative updater. Looks up a radius of 4 cells around the location indicated by the current flow estimate across all 4 pyramid levels, and feeds these values into a convolutional GRU to output "how much to correct the estimate" (Δ). The GRU with the same weights is repeated 12 times during training, and 32 times for Sintel / 24 times for KITTI during evaluation. The paper notes it did not diverge even when run up to 200 times.

Unlike previous methods (such as PWC-Net) that used pyramids in a staged coarse-to-fine structure, RAFT already knows pairs for all distances at a single ⅛ resolution, allowing it to handle fast-moving objects from the very beginning.

5. How It Is Trained

L = Σᵢ γ^(N−i) · ‖ fᵢ − f_정답 ‖₁,   γ = 0.8,  N = 반복 횟수

Loss is applied to the outputs of all iterations, with heavier weight given to later iterations (paper). This means early iterations are allowed to be quite wrong, but they must get progressively more accurate. Thanks to this loss, the GRU learns "correcting little by little" rather than "getting it right all at once."

When upsampling the ⅛-resolution flow back to the original resolution, instead of simple interpolation, the weighted average of neighboring 3×3 cells is calculated using network-learned weights (convex combination upsampling). This prevents flow from bleeding across object boundaries.

The training sequence was: from scratch (without pretraining) Flying Chairs 100k steps → FlyingThings 100k steps → a mix of Sintel, KITTI, etc., for 100k steps → KITTI 50k steps, trained using 2 desktop-grade GPUs (paper).

6. Hands-On Trial

code/05_flow_toy.py mimics the two core ideas of RAFT using only numpy.

1. A 64×64 image with smooth patterns is shifted by (3.4, −2.2) pixels to create the second frame (Fourier shifting ensures sub-pixel movement is accurate).
2. Using a 5×5 neighborhood patch around each pixel as features, the similarity of all pixel pairsbetween the two frames is computed at once into a table of 4,096×4,096 = 16.8 million entries (correlation volume).
3. The most similar location is selected within a radius of 8 cells around the starting point (0, 0) (lookup). This is accurate only to integer units.
4. Starting from that estimate, it repeats: "warp the second frame back according to the current estimate, and correct the remaining brightness difference divided by gradients." What RAFT's GRU does by learning , is replaced here by hand-coded Lucas–Kanade rules.

[무늬 있는 그림] 상관 부피 4,096×4,096 = 16.8M 칸
  반경 안 argmax(정수) 추정: EPE 평균 0.6546px · 중앙 0.4472px, 가장 흔한 값 (3, -2)
  갱신 1회: EPE 평균 0.1649px · 중앙 0.0414px
  갱신 2회: EPE 평균 0.1726px · 중앙 0.0602px
  갱신 5회: EPE 평균 0.1593px · 중앙 0.0499px

Comparing all pairs alone only achieves integer precision up to (3, −2), with a median error of 0.45px (the distance between the true values 3.4, −2.2 and the integers). A single update reduced this to 0.04px. The reason it didn't decrease further is the limitation of the hand-coded rules (bilinear interpolation errors when warping). This is why RAFT trains a GRU. Learning "how much to correct" from data instead of rules allows it to account for situations like interpolation errors or occlusion.

We tried the same thing on vertical stripe patterns.

[세로줄(구경 문제)]
  반경 안 argmax(정수) 추정: EPE 평균 7.5758px, 가장 흔한 값 (-4, 0)
  갱신 1–5회: 변화 없음 (식이 풀리지 않는 화소 100.0%)
  → 가로 움직임 중앙값 -4.00(참값 3.4), 세로 움직임은 17가지 값으로 흩어짐

Since vertical stripes look identical when moved up and down, vertical motion is impossible to determine (scattered across 17 values). Horizontal motion also cannot distinguish 3.4 from 3.4−8 = −4.6 because the stripe interval is 8 pixels (we picked the closest integer, −4). The update equations cannot be solved at all. This demonstrates the aperture problem numerically. In this case, learning models would also only be able to guess based on surrounding context (object boundaries, motion of other parts).

7. Paper Numbers

Fl-all is the percentage of pixels where the end-point error exceeds both 3 pixels and 5% of the ground truth magnitude. Lower is better.

SEA-RAFT modified RAFT's loss to "negative log-likelihood of a mixture of two Laplace distributions." It enables the network to output uncertainty itself, acting like L1 for easy-to-predict pixels while penalizing errors less for inherently ambiguous pixels (occlusions, aperture problem). It directly regresses the initial estimate instead of starting from 0, reducing the number of iterations (paper).

8. Limitations

• Computationally heavy. RAFT took around 0.5 seconds per 1080p frame (desktop-grade GPU, paper). This is an order of magnitude away from the 60fps per-frame budget (16.7ms).
• Weak on small, fast objects. At ⅛ resolution, a table tennis ball spanning a few pixels in diameter is smaller than a single feature cell. For ball motion, chaining ball detections is more accurate than optical flow (Part 9).
• Blur violates the brightness assumption. A blurred ball changes shape across frames, making it difficult to find the "same point."

9. Connection to Our Project

• Measurement: It was usable as a cheap and coarse signal. In 7 test matches of OpenTTGames (51,060 frames), we determined whether "a rally is currently in progress" on a 1-second basis using Farneback optical flow (a classical method) on video downscaled to 160×90. The AUC was 0.798 for player optical flow, 0.770 for video-wide frame differencing, and 0.860 for the BlurBall maximum heatmap value. Combining the ball and optical flow via rank averaging gave 0.876 , taking 2.5–2.8ms per frame on the CPU (repository docs/tt/PERFORMANCE.md 2.9.7). This means it can be used as an auxiliary signal to indicate "still in a rally" when the ball is occluded.
• There is still no evidence for bounce determination. Initial experiments simply concatenating the same optical flow and difference values as inputs to a learned bounce classifier (Part 4) showed no gains, and no conclusive results are available yet (as of the October 13, 2026 research log, ongoing research — the log is still outside the repository). Since only the "concatenation" combination method was tried, other combination methods remain open.
• Movement inside the ball tracker. TrackNetV4 and V5 in Part 9 use frame differencesinstead of optical flow as attention maps to highlight moving balls, while TOTNet variants that added optical flow to the input were more accurate during occlusion regions but correspondingly slower (Part 9).

A word from TT. When a striped shirt moves up and down, the computer doesn't know it moved. Seriously, we verified this directly with an example. That is why optical flow is used for "roughly where and how much things move," and the ball is found separately.

References

• Lucas & Kanade, An Iterative Image Registration Technique with an Application to Stereo Vision, IJCAI 1981.
• Farnebäck, Two-Frame Motion Estimation Based on Polynomial Expansion, SCIA 2003. https://doi.org/10.1007/3-540-45103-X_50
• Dosovitskiy et al., FlowNet: Learning Optical Flow with Convolutional Networks, ICCV 2015. https://arxiv.org/abs/1504.06852
• Sun et al., PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume, CVPR 2018. https://arxiv.org/abs/1709.02371
• Teed & Deng, RAFT: Recurrent All-Pairs Field Transforms for Optical Flow, ECCV 2020. https://arxiv.org/abs/2003.12039
• Wang, Lipson, Deng, SEA-RAFT: Simple, Efficient, Accurate RAFT for Optical Flow, ECCV 2024. https://arxiv.org/abs/2405.14793
• Our measurements: repository docs/tt/PERFORMANCE.md 2.9.7. Example code and outputs: code/05_flow_toy.py, code/05_flow_toy.out.txt