What Graph Optimization Changes: Count the Nodes

In one line

onnxruntime rewrites the graph when it opens a session. What changes and how far is decided by the optimization level, and you can check the result by taking it out as a file and counting the nodes.

Why this was needed

In an experiment comparing speed before and after quantization, the numbers just do not add up. Same model, same machine, yet it differs each time you measure, and sometimes the quantized side is slower.

The cause is often the optimization level. If one measurement ran with the default (everything on) and the other ran for some reason with optimization off, that comparison measured not quantization but optimization. A more common incident is saving the optimized file and deploying it — that file may not open outside the environment where it was made.

What the four levels do

The Graph optimizations documentation divides the levels into four.

• ORT_DISABLE_ALL: Does nothing. It runs the graph as it is.
• ORT_ENABLE_BASIC: Cleanups that do not change the meaning. It folds in advance the parts computed from constants alone (constant folding), removes nodes that are not needed, and groups common patterns into standard operators.
• ORT_ENABLE_EXTENDED: Fusions tailored to a specific execution provider are added. Here nodes of a non-standard domain appear.
• ORT_ENABLE_ALL: Layout transformations are added on top of this. The default is this level.

This description is what the documentation says, and how far it actually goes is decided by the model and the machine. So taking it out and counting is faster and more accurate than reading.

How to take it out and count

If you give a path to SessionOptions.optimized_model_filepath, the graph rewritten while opening the session is saved to that path. If you open that file with onnx.load and count the nodes, you can see exactly what happened.

원본                 Add(상수,상수) MatMul Add Relu Identity Mul   노드 6개
ORT_DISABLE_ALL      Add MatMul Add Relu Identity Mul             노드 6개
ORT_ENABLE_BASIC     Gemm Relu Mul                                노드 3개
ORT_ENABLE_EXTENDED  FusedGemm Mul                                노드 2개

Three things are visible here at once. The node that added constants together disappears and its result settles in as an initializer. Identity is removed because it is not needed. The Add after MatMul is grouped into a single Gemm, and in the next level it swallows even Relu to become FusedGemm.

The answer does not change just because the nodes decreased. You can confirm this by feeding the same input through the four levels and comparing the results. However, float32 has only 7 significant digits, so if a fused kernel changes the order of multiplication, the last digit can wobble. That is why you ask not "is it exactly the same" but "is it within a tolerance you have set".

What it looks like in the field

First, deploying the optimized file. The extracted file is small and has few nodes, so it seems like "I should send this". But that file contains nodes of a non-standard domain such as com.microsoft. Another runtime that knows only standard operators cannot open that file. ORT itself also issues a warning when saving that it should be used only in the same environment.

Second, onnx.checker passes that file. That is because the checker sees an unknown domain as someone's extension and moves on. So if you use "checker passes" as the basis for portability, you cannot catch this incident. The basis is the list of the nodes' domains.

Third, not aligning the comparison baseline. To measure the effect of quantization, the two measurements must be at the same optimization level. If you fix the baseline at ORT_DISABLE_ALL, it does not wobble. And the numbers that will go into real operation must be measured separately at the same level as operation.

Fourth, reading the node count as performance. Fewer nodes does not necessarily mean faster. When emulation gets involved, as on a Mac like this one, the time wobbles by up to a factor of two. That is why this lab does not measure time — it judges only how the structure changed and whether the answer is the same. Time is measured separately on the same machine as operation.

What really matters in practice

• Take it out and count. Do not guess from the documentation; take the optimized graph out as a file and count the nodes.
• Also measure whether the answer is the same. Set a tolerance and write down its basis.
• Judge portability by domain. The checker passing is not a basis.
• Send the original for deployment. Let the receiving side do the optimization in its own environment.

What you will do in the next lab

You gather constant folding, a node that is not needed and three nodes that will be fused into one graph and build it yourself, and grow the tool optlevel.py one step at a time. You run all four levels and set the node counts side by side, write down what disappeared and what appeared at each level, and feed the same input, made with a seed the grader sets, through the four levels to measure whether the answer is the same. At the end you judge portability from the domain list of the extracted file. The grader builds its own graph each time with different shapes, weights and seeds, actually runs your tool, and checks the answers against its own.