Batching and Latency: Batch Size, Concurrent Sessions, and GPU Sharing

Status: Draft (2026-10-16), partially measured. Latency per batch was measured as one-off pods on a single home lab 8GB laptop GPU (no other GPU jobs, median of repeats after warm-up). Session simulation is based on laptop CPU calculations inserting that table (code/06_batching_sim.py, around 1 second). Actual concurrent session values for workers cited load test data from the TT Score performance documentation (server GPU, at a time when cards were shared with other jobs). Part 6 of the "Reading Vision Models 3" series. The distinction between latency and throughput is in Part 1.

Table of Contents

• 1. Intuition: Buses and Taxis
• 2. Server-Side Research: Batching within Latency Targets
• 3. Direct Measurement: Does Batching Make the Ball Model Faster?
• 4. When Multiple Sessions Share a Single GPU
• 5. Methods for GPU Sharing
• 6. Limitations
• 7. Connection to Our Project
• References

1. Intuition: Buses and Taxis

A GPU is like a bus with thousands of computational units. If it departs with just one passenger (input), most seats remain empty; if it waits to gather several passengers, it carries many at once. However, earlier passengers have to wait while others are being collected. Batching is a trade-off where you spend latency to gain throughput. Moreover, if the bus is already full (meaning a model whose single input fully utilizes all compute units), gathering more brings no benefit.

2. Server-Side Research: Batching within Latency Targets

• Clipper (Crankshaw et al., 2017) automatically finds the "largest batch that does not exceed the latency target" for each model. It gradually increases the batch size until the target is exceeded, then reduces it by 10% (AIMD). Under a 20ms target, throughput increased by up to 26x, though this was for a very lightweight model, a CPU linear SVM (Section 4.3). "Delayed batching," which waits up to 2ms for the batch to fill, increased throughput by 3.3x for the same model, but offered no benefit for models where even small batches were already efficient.
• Clockwork (Gujarati et al., 2020) looks at the other side. The inference time of a single DNN is inherently very consistent; on a datacenter GPU, the 99.99th percentile of batch 1 was within 0.03% of the median (11 million runs, Section 2). What created tail latency was upstream choices like running multiple inferences simultaneously; concurrent execution increased throughput by up to 25% while multiplying tail latency by 100. Therefore, they ran only one at a time per GPU and scheduled them centrally, effectively meeting a 100ms target in all cases (99.997% of requests according to the conference presentation page, while the arXiv abstract states 99.9999%, which differ).
• NVIDIA Triton's Dynamic Batcherprovides this as a configuration knob. By default, it does not wait, grouping "whatever is currently in the queue" up to the maximum batch size; if you specify max_queue_delay_microseconds , it waits up to that duration to fill the batch. The documentation recommends the following order: "determine max batch → enable with default values → measure and increase or add wait time if within the latency budget."

Combining these three yields a principle: A latency budget is established first, and batch size is only increased within that budget. Furthermore, how much of a gain it provides must be measured on that specific model and GPU.

3. Direct Measurement: Does Batching Make the Ball Model Faster?

We measured TT Score's ball model (BlurBall, 3-frame window, 1.49M parameters) on an experimental GPU by varying only the batch size (512x288, median of 3 batches of 30 runs after 10 warm-up runs, cuDNN auto-tuner enabled).

Throughput decreased as batching increased. Because Batch 1 was already fully utilizing this GPU, batching only enlarged intermediate tensors (since the model maintains 512x288 resolution throughout, a single FP32 feature map is tens of megabytes), which likely caused cache overflow (probable cause). It is not because the GPU heated up and slowed down from prolonged running; measuring Batch 1 again after running the heaviest workload (1024x576 batch 4) for a few minutes yielded the same initial value (9.77 vs 9.82ms).

Reducing the input to 256x144 reverses this result.

Because a single small input leaves the GPU underutilized, batching four together cut the inference time in half in FP16. Even for the same model, the pros and cons of batching reverse depending on the input size and the GPU. When measured on a server GPU (24GB class) with the same model on 2026-10-12, a batch size of 3 yielded a slight benefit, taking 0.83x the inference time (FP16 8.0ms → batch size 3 20.0ms) (TT Score Performance Document 2.9.8).

4. When Multiple Sessions Share One GPU

A real-time worker receives "sessions" instead of "requests." Frames arrive at 30fps per session, and the worker holds only the newest frame for each session (the "latest-only" policy from Part 1). When the GPU is free, the TT Score worker's approach is to batch together the latest windows of all waiting sessions. We simulated this by session count using the empirical tables above (GPU share only, 30 seconds).

(code/06_batching_sim.out.txt. (Since the server GPU table only has two data points, batch sizes of 4 and above are assumed to scale linearly.)

Here is how to read this.

• When the GPU has spare capacity (session 3 or fewer), batching makes no difference. Frames rarely arrive simultaneously, so batches almost never form.
• When the GPU is under-provisioned, batching buys fairness. "Oldest frame first" continually pushes back sessions whose frame timestamps keep growing due to overwriting, starving individual sessions down to 2–5fps. Batching ensures all sessions receive the same fps.
• Throughput follows the batch curve. On GPUs that benefit from batching (server GPU table), session 5 achieved a steady 30fps across the board, whereas on GPUs that do not benefit (laptop GPU), more batching reduced the total throughput and increased latency.

An actual load test of the worker (server GPU, sharing cards with other workloads on 2026-10-11) showed the same pattern. A single session achieved 29.4fps with a round-trip p99 of 32ms; three sessions yielded 16–20fps per session with a round-trip median of 70–95ms, and 21–30% of sent frames were dropped being overwritten by newer frames (Performance Document 4.2).

5. Methods for Sharing a GPU

There are three ways for multiple workloads to share a single GPU (NVIDIA GPU Operator and Device Plugin documentation).

Time-slicing may seem free, but if another workload on the same card is busy, your latency increases directly. This appears to be what Clockwork meant by "concurrent execution enlarges the tail" in production. In TT Score's actual measurements, a single inference of the ball model took 10.5ms when the card was idle, but 34ms when running alongside 2–3 other tasks (Performance Document 2.9.8). Real-time latency is determined more by "who else is sharing the card" than by the model itself.

6. Limitations

• Batch measurements cover only one GPU type and one model. The root cause of reduced throughput (cache overflow) is an assumption and was not verified with a profiler.
• The session simulation lacks network, JPEG decoding, and post-processing. Because actual workers perform these tasks on separate threads, the behavior should be similar when the GPU share is the bottleneck, but the exact numbers will differ.
• The starvation in "oldest frame first" is an artifact of this simulation's rules. A round-robin schedule would prevent starvation. It was not compared directly against batching (unmeasured).

7. Connection to Our Project

• The real-time worker combines "batching all latest windows of waiting sessions" with "dropping backlogged windows." Batching provides fairness even when throughput gains are small, while dropping preserves the latency upper bound. The concurrent session limit (3) also derives from this latency budget.
• The results in Section 3 serve as a warning that "even with the same code, changing the card alters the pros and cons of the batching policy." If a worker is moved to a different card, the batch curve must be re-measured.
• As long as the card is shared with other tasks like TTS via time-slicing, 30fps cannot be guaranteed for all three sessions. This decision (whether to reduce concurrency or split the card) has been documented with figures in the performance document.

The next post shifts focus from servers to on-device execution on phones.

A word from TT. If a bus is already full, letting more people on just delays closing the doors. My ball model was exactly like that—the more we batched, the slower it got!

References

• Crankshaw, D. et al. (2017). Clipper: A Low-Latency Online Prediction Serving System. NSDI 2017. https://arxiv.org/abs/1612.03079
• Gujarati, A. et al. (2020). Serving DNNs like Clockwork: Performance Predictability from the Bottom Up. OSDI 2020. https://arxiv.org/abs/2006.02464
• NVIDIA Triton Inference Server — Dynamic Batcher. https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/user_guide/batcher.html
• NVIDIA GPU Operator — Time-Slicing GPUs in Kubernetes. https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-sharing.html
• NVIDIA k8s-device-plugin README(Shared Access to GPUs). https://github.com/NVIDIA/k8s-device-plugin
• Our Measurements: Empirical data and repository of this series docs/tt/PERFORMANCE.md 2.9.8·4.2
• Example Code and Outputs: In this folder code/06_batching_sim.py, code/06_batching_sim.out.txt