The 33ms Budget: End-to-End Latency and Throughput Are Different Numbers

Status: Draft (2026-10-16), partially measured. Step-by-step times were measured on a server equipped with a single 8GB laptop-class GPU using disposable pods (capped at 4 CPU cores) (median of 300 runs after 30 warm-up runs). Capture and encoding times on the phone, as well as the internet transmission segment, are unmeasured. The queue simulation is a laptop CPU example (code/01_latency_queue.py, around 1 second). Part 1 of the series “Reading Vision Models 3: Running in Real-Time”. This series covers how to run the models seen in Reading Vision Models 1·2within a fixed time limit .

Table of Contents

• 1. The Two Numbers of “Real-Time”
• 2. Overlapping Steps: Increased Throughput, Unchanged Latency
• 3. When Overloaded: Queue Everything or Drop?
• 4. A Late Correct Answer Is a Wrong Answer: Streaming Perception
• 5. Our Measurement: The Life of a Single Ball-Finding Frame
• 6. The Habit of Reading Numbers
• 7. Limitations
• 8. Connection to Our Project
• References

1. The Two Numbers of “Real-Time”

For a table tennis scoreboard app to be considered “real-time”, two conditions must be met simultaneously:

• Throughput: How many frames are processed per second. If the camera provides 30fps, throughput must be 30 or higher to avoid falling behind.
• End-to-end latency: From the moment a frame is captured until the result appears on the screen. This determines when the scoreboard updates after a ball bounces.

The two are frequently confused but are distinct numbers. Throughput is determined by the “slowest single step”, while latency is the “sum of all steps + time spent waiting in the queue”. The 33.3ms frame interval at 30fps (1000 ÷ 30 = 33.3ms ) is a budget for throughput, not an upper bound for latency. By overlapping steps, you can achieve 30fps even if the latency exceeds 33ms; conversely, even with a 10ms latency, if a queue builds up, you end up viewing results from a second ago.

2. Overlapping Steps: Increased Throughput, Unchanged Latency

If preprocessing takes 8ms and inference takes 25ms sequentially, it takes 33ms per frame, yielding 30.3 frames per second. If preprocessing is performed in advance on a separate thread (pipelining), the GPU can continuously run inference without idling, pushing throughput up to 40 frames per second. However, the time a single frame experiences remains 8 + 25 = 33ms.

처리량  = 1 ÷ (가장 느린 단계의 시간)
지연    = 단계 시간의 합 + 줄에서 기다린 시간
리틀의 법칙: 처리 중인 프레임 수 = 처리량 × 지연   (40장/초 × 0.033초 = 1.32장)

Little's Law (a fundamental formula in queueing theory) can also be applied in reverse. If throughput is fixed but the number of “frames currently being processed or waiting” increases, it means latency has increased proportionally.

3. When Overloaded: Queue Everything or Drop?

Inference time varies slightly every time. While an average of 25ms usually stays within budget, what happens when the average exceeds the budget at 36ms? We simulated 60 seconds of a 30fps camera (code/01_latency_queue.out.txt, where inference time follows a log-normal distribution and numbers are hypothetical for illustrative purposes).

Within budget, both policies behave identically. When the budget is exceeded, queuing maintains throughput nearly intact while latency accumulates indefinitely(displaying a screen lagging by 4.5 seconds after a minute). “Latest Only” grabs only the newest frame whenever the processor becomes free and discards delayed ones. Although it drops 8.6% of frames, latency is bounded. Since results from frames a few seconds ago are useless in real-time video, dropping frames is the correct choice. TT Score's real-time workers and browsers also operate this way (Part 8).

4. A Late Correct Answer Is a Wrong Answer: Streaming Perception

Li, Wang, and Ramanan's “Towards Streaming Perception” (ECCV 2020) formalized this concept into an evaluation metric. Standard detection accuracy (AP) assumes the world waits for the model. The paper's streaming AP compares the ground truth at time t against the latest output whose computation

(Paper Table 1 and 2, one top-tier consumer GPU from 2017.) Three things are visible.

• An accurate but slow model collapses in real-time (38.0 → 6.2).
• That doesn't mean the fastest model wins either. Somewhere in the middle is optimal.
• Prediction(pushing forward by as much as the object would have moved during the latency), a slower but more accurate model wins again. If you cannot "reduce" latency, you can "bridge" it.

"Scheduling" is interesting. If the next frame is likely to arrive soon, we wait instead of starting the current frame. This is better than missing a new frame because we were clinging to an old one. It is the same spirit as "Latest Only" in Section 3.

5. Our Measurements: The Life of a Single Ball-Tracking Frame

We measured the real-time pipeline of TT Score step by step on the server side. A single actual match screen (1920×1080 reduced to camera size 1280×720) was run 300 times. The phone's tasks (resizing, encoding) were also simulated on the same server CPU.

The time spent on a single frame on the server is about 19ms (FP32), of which inference accounts for 80%. Among the remaining portions, the largest ones were normalization (1.35ms) and transferring to GPU (1.76ms) . Converting 512×288×3 to float32 creates 1.77MB per frame, or 5.3MB for three windows, which is created and transferred every time. If you send uint8 as-is to the GPU and have the GPU perform the normalization, the transfer volume becomes a quarter and the GPU also handles the normalization. This has not been changed yet, and its effect is unmeasured.

This table omits phone camera capture, actual encoding, internet round trip, web server relay, and screen rendering. Those parts are summarized together with end-to-end values measured via actual browser tests in Part 8.

6. Habits for Reading Numbers

• Look at the median and the tail together. What users perceive as "stuttering" is closer to the p99 end. In Section 3, even within the budget, p99 was 44ms, exceeding the budget.
• Carry the conditions together. "10ms inference" can vary up to 3fold depending on which GPU, what batch size, and whether other tasks are running concurrently (Part 6).
• Measure after warm-up. The first execution of a GPU takes anywhere from a few seconds to longer due to kernel selection and memory allocation. All measurements in this series are the median of repetitions after warm-up.

7. Limitations

• The simulation in Section 3 is a single-stage inference, and the time distribution is an assumption. Real systems have multiple stages, and each stage has queues.
• Section 5 only measured the server side. How long the same resizing and encoding take on the phone CPU is unmeasured.
• Primary data on at what level of delay humans perceive something as "late" was omitted because we could not verify the original text this time.

8. Connection to Our Project

• In table tennis, a "late correct answer" is when the scoreboard changes after the next rally has already started. TT Score aims to deliver judgments within 1.5 seconds after the end of a rally and is measuring accordingly (Part 8).
• The browser pauses sending if there are 3 frames without a received answer, and the worker overwrites the backlog window with a new one. It is effectively doing Section 3's "Latest Only" at both ends.
• The subsequent posts in this series are ways to reduce the largest cell in the table (inference): Precision (Part 2), Compiler (Part 3), Smaller Model (Part 4), Running Less Frequently (Part 5), Sharing Among Multiple Users (Part 6), and Running on the Phone (Part 7).

TT's Word. If you shout "Bounce!" 4 seconds after the ball bounces, no matter how accurate you are, I'm already hitting the next ball. A late correct answer is a wrong answer!

References

• Li, M., Wang, Y.-X., Ramanan, D. (2020). Towards Streaming Perception. ECCV 2020. https://arxiv.org/abs/2005.10420
• Our Measurements: Empirical measurements in this series (raw data is in out-of-repository research logs), repository docs/tt/PERFORMANCE.md Section 4
• Example Code and Output: In this folder's code/01_latency_queue.py, code/01_latency_queue.out.txt