Graph Compilers and Runtimes: Operator Fusion, torch.compile, ONNX Runtime, and TensorRT

Status: Draft (2026-10-16), partially measured. torch.compile, memory layout, and cuDNN settings were measured using single-use pods on a single 8GB laptop-grade GPU in the lab (PyTorch 2.9, median of 50 runs × 5 batches after 20 warm-up runs). The principles of fusion were verified using a laptop CPU example (code/03_fusion.py, taking several seconds). ONNX Runtime and TensorRT were not measuredas they were absent from the experimental image (not downloaded). Part 3 of the "Reading Vision Models" series. Precision reduction is covered in Part 2.

Table of Contents

• 1. Intuition: The Chef and the Refrigerator
• 2. Try It Yourself (Laptop CPU)
• 3. What Graph Compilers Do
• 4. ONNX and Runtimes: Portable Model Formats
• 5. Our Measurements: Applying torch.compile to the Ball Model
• 6. Limitations
• 7. Connection to Our Project
• References

1. Intuition: The Chef and the Refrigerator

While a GPU's compute units are fast, data resides in memory outside the chip (the refrigerator). Eager execution performs "convolution → batch normalization → ReLU → addition" separately. The convolution result goes into the refrigerator, batch normalization takes it out, modifies it, and puts it back, and then ReLU takes it out again and puts it back. For large feature maps, the time spent moving data in and out exceeds the computation time. Operator fusionmeans "taking it out once, doing all four operations, and putting it back only once."

There are two types of fusion:

• Pre-folding values: During inference, batch normalization (BN) consists of channel-wise fixed multiplications and additions, which can be folded directly into the preceding convolution weights and biases. The operation disappears entirely.
• Kernel chaining: Element-wise operations like ReLU, addition, and type casting are chained together with preceding and succeeding kernels into a single kernel.

BN 접기:  y = γ·(W∗x + b − μ)/√(σ²+ε) + β
        = (γ/√(σ²+ε))·W ∗ x + (γ·(b − μ)/√(σ²+ε) + β)
        = W' ∗ x + b'           ← 합성곱 하나로

2. Try It Yourself (Laptop CPU)

code/03_fusion.py was run using a single laptop CPU thread (median of 7 runs).

• Folding BN yielded mathematically identical results, but did not reduce time on this CPU. A single BN next to a large convolution constituted a small share (it decreased by 2.6% on the first run, and 0% on the second—within the noise floor).
• Element-wise chains are different. Eager execution reads and writes large arrays four times, whereas the compiled version reads once and writes once. The computation is identical, yet it became 4 times faster simply by reducing memory round-trips. On this machine, the first compilation took 5.4 seconds without a cache, and 0.6 seconds with a cache.
• Even when using manual in-place operations, it only reached 7.4ms. While it reduced new array allocations, it still read and wrote four times. There are limits to optimizations humans can perform at the Python level, and significant reductions require descending one layer lower (to kernels) .

3. What Graph Compilers Do

TVM (Chen et al., 2018) receives a model as a graph and classifies operations into four categories: element-wise (injective), reduction, complex operations that can attach element-wise work to the output (complex-out-fusable, such as convolution), and unfusable operations. It groups the first three according to rules. Operator fusion alone provided a 1.2–2x speedup (2015 desktop-class GPU, 4 tasks including convolution+BN+ReLU, Section 3, Figure 4), and achieved 1.6–3.8x end-to-end compared to other frameworks (Section 6.1).

PyTorch 2 (Ansel et al., 2024) 's torch.compile consists of two parts:

• TorchDynamo: Intercepts Python bytecode to capture computation graphs. If it encounters Python code that cannot be converted into a graph, it breaks the code around it and runs the rest as-is. Graph capture overhead was about 5% in inference (Table 2, datacenter GPU).
• TorchInductor: Fuses the captured graph to generate Triton kernels for GPUs (or C++ for CPUs).

On datacenter GPUs, inference for 62 TIMM image models was 2.48x faster in FP32 and 2.77x faster in FP16 compared to eager execution (Table 3). Disabling both fusion and inlining actually made it slower at 0.80x relative to eager execution (Table 4, HuggingFace FP16 inference). This means most of the speedup comes from fusion. Since we could not verify in the text which combination the green "2.27x inference" refers to, we rely only on the values from Tables 3 and 4 here.

4. ONNX and Runtimes: Portable Model Formats

torch.compile runs within PyTorch. If you do not want to keep Python and PyTorch on the server, or if you need to run the model in a browser or phone, you must export the model into a standard graph file.

• ONNX: An open format specifying operator types and graphs. Once exported from PyTorch, other runtimes can read it.
• ONNX Runtime: A runtime that executes ONNX models. It performs graph optimizations in three stages: basic (constant folding, eliminating redundancies like convolution+BN), extended (complex fusions like GELU and attention, for CPU/CUDA, etc.), and layout (adjusting memory layouts to suit the CPU). Default settings enable all of these (official documentation). Actual computation is handled by execution providers—such as CPU, CUDA, TensorRT, Core ML, and browser-based WebGPU/WASM. The value of ONNX lies in carrying the same file across multiple devices (Part 7).
• TensorRT: An NVIDIA GPU-exclusive optimizer. It fuses layers, empirically benchmarks kernel candidates to select the fastest ones, and reduces precision to FP16 or INT8. The result is an "engine" file tailored to that specific GPU, meaning moving it to a different GPU requires rebuilding it. Previously, it used a "calibrator" (entropy, min-max) to determine the INT8 range using calibration data, but as of TensorRT 10.1 in the official documentation, calibrators have been deprecated in favor of an explicit approach that embeds quantization and dequantization nodes directly inside the model.

5. Our Measurements: Applying torch.compile to our Ball Model

We measured TT Score's ball model (BlurBall, 3-frame window, 512×288, batch size 1) on our experimental GPU.

(FP16 uses autocast. Multipliers are relative to eager execution at the same precision.)

Profiling the same execution revealed that a single eager FP32 run invoked 478 CUDA kernels, and out of 14.88ms of active GPU time, convolutions took 8.09ms (72 times) and batch normalization took 3.27ms (71 times, 22%). Since the sum of kernel times roughly matched the wall-clock time (15.09ms), the GPU was never idle. Therefore, the bottleneck was not dispatch overhead, but memory-bound element-wise operations, and we interpret that the compiler eliminated these by fusing them (the number of kernels in the compiled version was not counted).

Here is how to read these results.

• FP16 achieved greater gains. In FP16 eager execution, convolutions became faster, but BN, ReLU, and type casting remained as-is, increasing their proportional share—and the compiler wiped out precisely those operations. Precision reduction from Part 2 and fusion from Part 3 multiply with each other (15.09 → 4.34ms, 3.5x).
• CUDA graphs (reduce-overhead) provided a modest additional reduction. By wrapping a single execution's kernels into a graph and dispatching them to the GPU all at once, Python and driver dispatch overhead per kernel is eliminated. Since this model already kept the GPU busy, the benefit was minor (4.61 → 4.34ms).
• Longer tuning did not yield higher speeds. max-autotune took 45 seconds and was not faster than the default. A warning also appeared stating that the compute units on this GPU were too small to benefit from matrix multiplication auto-tuning.
• Memory layout (NHWC) only benefited FP16(9.77 → 9.02ms) and caused a slowdown in FP32 (15.09 → 18.62ms). cuDNN auto-tuning made no difference. Measuring on that specific model and precision is more accurate than relying on the general rule of thumb that "this setting is faster."

6. Limitations

• The first call takes a long time. Even the default mode takes 10 seconds. Real-time workers can hide this by warming up upon startup, but if the input shape (batch size) changes, it may trigger re-compilation. All batch sizes (1, 2, and 3) must be warmed up (Part 6). Persisting compilation caches to disk to minimize restart times was not tested (not measured).
• Output correctness must be verified independently. Because fusion alters the order of addition, the least significant digits may change. Whether ball positions remain identical was not measured this time (not measured). Just like the FP16 comparison in Part 2, it should be enabled after verifying with ground-truth frames.
• ONNX Runtime and TensorRT could not be measured. It was not in the experiment image, and no new packages were downloaded in this task. Whether TensorRT's FP16 and INT8 engines are faster than torch.compile is not answered in this post.
• These are the results for a single GPU and a single model. For models centered around large matrix multiplications like Transformers, the share of fusion may differ.

7. Connection to Our Project

• For operations, the current execution is FP32. According to Section 5, using FP16 + compile reduces a single instance from 15ms to 4–5ms, creating headroom to accept more sessions on the same GPU or share it with other tasks. Two things must be done before turning it on: verify that the ball position and judgment match using ground truth frames from recorded matches (Method in Part 2), and re-benchmark on the production GPU (gains and losses change if the GPU changes — Part 6).
• During startup, the operator warms up batches 1, 2, and 3. When compilation is enabled, that warmup becomes the compilation itself, increasing preparation time by tens of seconds. Because the readiness check (/healthz) opens after the warmup, users will not notice that time.
• Models to be run on phones or browsers should first be exported to ONNX (Part 7).

A word from Titi. We cooked the exact same dish while opening the refrigerator door less often, and it became twice as fast—even though the ingredients (weights) and recipe (model) stayed the same!

References

• Chen, T. et al. (2018). TVM: An Automated End-to-End Optimizing Compiler for Deep Learning. OSDI 2018. https://arxiv.org/abs/1802.04799
• Ansel, J. et al. (2024). PyTorch 2: Faster Machine Learning Through Dynamic Python Bytecode Transformation and Graph Compilation. ASPLOS 2024. https://doi.org/10.1145/3620665.3640366
• ONNX. https://onnx.ai/ · ONNX Runtime Graph Optimization. https://onnxruntime.ai/docs/performance/model-optimizations/graph-optimizations.html
• NVIDIA TensorRT Documentation (How TensorRT Works, Explicit Quantization). https://docs.nvidia.com/deeplearning/tensorrt/latest/architecture/how-trt-works.html
• Our Measurements: Actual measurements from this series (raw data in out-of-repository research logs).
• Example code and output: In this folder's code/03_fusion.py, code/03_fusion.out.txt