Building Smaller Models: Depthwise Separable Convolution, Pruning, Distillation, and EfficientTAM

Status: Draft (2026-10-16), partially measured. Direct hands-on work includes a laptop CPU example (code/04_small_models.py, around 1 second) and EfficientTAM measurements on an experimental GPU (Section 6, a single 8GB-class laptop GPU in the home lab, ephemeral pod). Paper figures are transcribed from the original texts along with their conditions, and hardware is noted by tier. Part 4 of the series 「Reading Vision Models 3」. The learning principles of distillation are in Reading Vision Models 2, Part 4, and the architecture of EfficientTAM is in Reading Vision Models 1, Part 10.

Table of Contents

• 1. What to Reduce: Computation, Parameters, and Memory Access
• 2. Changing the Architecture: Depthwise Separable Convolution (MobileNet)
  • Hands-on Measurement: If Computation Drops by 8.4x, Does Time Drop by 8.4x?
• 3. Scaling Uniformly: EfficientNet
• 4. Trimming Down: Pruning
  • Hands-on Measurement: If 90% is Zero, Is It 10x Faster?
• 5. Transferring What Was Learned: Distillation
• 6. Example: EfficientTAM, a Downsized SAM 2
  • Our Measurement: EfficientTAM-S on an Experimental GPU
• 7. Limitations
• 8. Connection to Our Project
• References

1. What to Reduce: Computation, Parameters, and Memory Access

The phrase "making a model smaller" conflates three different things.

• Number of multiply-accumulates (MACs, roughly half of FLOPs): The amount of computation performed in a single inference.
• Number of parameters: The download size and the size to load into memory.
• Memory access volume: The bytes read and written to perform computations. As seen in "The Life of a Frame" in Part 1, this is often what determines actual time.

These three do not move together. Most measurements in this article demonstrate this mismatch.

2. Changing the Architecture: Depthwise Separable Convolution (MobileNet)

A standard 3x3 convolution looks at the 3x3 neighborhood of all input channels for every output channel produced. It effectively performs "spatial viewing" and "channel mixing" all at once. MobileNet by Howard et al. (2017) separates these two.

1. Depthwise 3x3: A single 3x3 filter per channel. It looks only at spatial areas.
2. Pointwise 1x1: Mixes channels at the same position.

Given input channels M, output channels N, filter side K, and feature map size HxW,

보통 합성곱        H·W·M·N·K²
깊이별 분리        H·W·M·K²  +  H·W·M·N
비율(분리 ÷ 보통)  1/N + 1/K²

. When K=3 and N is large, this is roughly one-ninth. The paper states that this is 8 to 9 times less computation than a 3x3 (Section 3.1).

(MobileNet paper, Table 4.) Giving up 1%p of accuracy reduced computation by about 8.6x. The fact that 95% of computation is concentrated in the 1x1 (Table 2) is also by design, because 1x1 convolutions are simply matrix multiplications and run fast on any hardware.

MobileNet provides two knobs for sizing. The width multiplier α(multiplied by the number of channels) reduces computation and parameters roughly by α², while the resolution multiplier ρ reduces only computation roughly by ρ² (parameters remain unchanged). In Tables 6 and 7 of the paper, α=0.5 yielded 70.6% → 63.7% (149M MACs), and an input size of 128 yielded 64.4% (186M MACs). As we will see in Part 5, reducing resolution pays a different price: "small objects disappear."

MobileNetV2 (Sandler et al., 2018) uses an inverted residualthat expands the narrow bottleneck by 6x using a 1x1, applies a depthwise 3x3, and then narrows it back down. The final 1x1 that narrows it back down does not have ReLU applied (based on the observation that applying ReLU in low-dimensional spaces crushes information). On a single large phone CPU core, V1 was 113ms → V2 75ms, with top-1 going from 70.6% to 72.0% (Table 4).

Hands-on Measurement: If Computation Drops by 8.4x, Does Time Drop by 8.4x?

With 128 channels and a 72x128 feature map (one-fourth the size of a 512x288 input), standard 3x3 and depthwise separable convolutions were measured using a single laptop CPU thread (median of 9 runs, code/04_small_models.out.txt).

The computation decreased by 8.4x, but the time decreased by only 1.6x. The culprit is the depthwise 3×3 convolution. It accounts for only 7% of the computation, yet it took up 79% of the time. Since depthwise convolution does not mix channels, there is very little computation performed per loaded value (low arithmetic intensity). Matrix-multiplication-dedicated hardware is only fast at "performing many calculations with values loaded once," so depthwise convolutions leave that hardware mostly idle. Because this laptop CPU has a matrix accelerator that makes dense convolutions exceptionally fast, the gap may have appeared larger (other CPUs and GPUs were not measured).

ShuffleNet V2 (2018) by Ma et al. is a paper that directly addresses this phenomenon. Structures with similar computational costs ran faster in different orders on GPUs and mobile CPUs, leading to the conclusion that "computational cost is merely a proxy metric for speed; measure directly on the target device." Two of the four guidelines derived from it align with the results above: do not overuse group convolutions (increasing the number of groups from 1 to 8 while keeping computational cost constant drops desktop-class GPU throughput from 2,451 to 634), and do not ignore element-wise operations (ReLU and addition) (removing both from a ResNet block makes both GPUs and mobile CPUs about 20% faster) (Tables 2 and 4).

3. Growing Evenly: EfficientNet

EfficientNet (2019) by Tan and Le asks the opposite question. When you have a small baseline network, what should you scale: depth, width, or resolution? Since scaling just one quickly leads to saturation, all three are scaled together at fixed ratios.

깊이 d = α^φ,  폭 w = β^φ,  해상도 r = γ^φ,   조건 α·β²·γ² ≈ 2   (α, β, γ ≥ 1)

Because the computational cost of convolution is proportional to depth and scales quadratically with width and resolution, every increase of φ by 1 roughly doubles the computation. The baseline network B0 was found via neural architecture search with α=1.2, β=1.1, and γ=1.15.

(Paper Table 2.) In the abstract's statement, "8.4x smaller and 6.1x faster than the best models at the time," "faster" refers to the latency on a single server CPU core with batch 1 (19.0s vs. 3.1s, Table 4). The fact that this is not about GPU throughput is often omitted in summaries.

4. Trimming Down: Pruning

This is a method where small weights are zeroed out in a trained network, followed by a brief retraining to recover accuracy. Deep Compression (2016) by Han et al. reduced AlexNet from 240MB to 6.9MB (35x) and VGG-16 from 552MB to 11.3MB (49x) while maintaining accuracy through pruning → weight quantization (8-bit for convolutions, 5-bit for fully connected layers) → Huffman coding (Table 1). Pruning alone reduced the number of connections by 9x for AlexNet and 13x for VGG-16.

Here, a distinction must be made between "smaller" and "faster." The 3–4x speedup reported in this paper was measured only for fully connected layers with batch 1 , and the paper itself notes that the advantage disappears with larger batches (Section 6.3, Appendix A).

Measuring Directly: If 90% Are Zeros, Is It 10x Faster?

We measured the time it takes to multiply 64 inputs by a 2048×2048 fully connected layer (laptop CPU, single thread, median of 9 runs).

Unstructured pruning keeps the shape intact, so dense multiplication multiplies zeros as well. Converting to a sparse format skips zeros, but at this size and sparsity, due to irregular access costs, it was 11.6x slower. Only structured pruning, which removes entire rows, resulted in a smaller dense multiplication and became about 10x faster. This is why Li et al. (2017) proposed a method to remove entire convolution filters. It reduced computational cost by 34% in VGG-16 on CIFAR-10, and inference time on a desktop-class GPU (batch 128) dropped from 1.23s to 0.73s (Tables 1 and 3). Unstructured sparsity could also become faster with dedicated kernels or hardware, but that was not measured (unmeasured).

5. Transferring What Was Learned: Knowledge Distillation

Training a small architecture from scratch often fails to learn as much as a large model. Knowledge distillationuses the soft outputs of a large model (teacher) as target values for a small model (student) during training. The principles and figures are detailed in Reading Vision Models 2, Part 4. From a runtime perspective, there is one key takeaway: Distillation does not change the architecture. The student's speed is determined by the student's architecture (Sections 2–4), and distillation simply helps that architecture approach its maximum possible accuracy.

6. Example: EfficientTAM, Which Reduced SAM 2

EfficientTAM (2024) by Xiong et al. is an example of applying the above methods together to a video segmentation model. Reading Vision Models 1, Part 10covers the architecture, so only the trimmed parts are revisited here.

• Replacing the encoder with a small, flat ViT: ViT-Tiny/Small instead of hierarchical Hiera. This changes the architecture (Section 2).
• Pretraining by mimicking large model features: EfficientSAM's SAMI pretraining (reconstructing SAM encoder features from masked patches)—a broad form of distillation (Section 5).
• Averaging memory to 2×2: Reducing the keys and values of memory cross-attention to one-fourth. This approach reduces computation (same spirit as Section 4).

(1024×1024 input, GPU batch size 1. The phone is a 2023 high-end model, and measurement precision and input size are not specified in the paper.) In Table 5, where the input was reduced to 512×512, the phone latency was 80.6ms and SA-V was 71.5. As pointed out in part 10 of reading vision models, numbers close to the abstract's "approx. 10 FPS on phone" apply only to this 512 input version.

Our Measurement: EfficientTAM-S on an Experimental GPU

TT Lab's ping-pong table detector uses EfficientTAM-S. We measured the same weights (verified via sha256) on a single 8GB laptop-grade GPU in our lab. Tested using ephemeral pods, no other GPU workloads, taking the median of 10 runs × 3 batches after 5 warmup runs.

• The parameter count is 34.06 million in total, of which 22.35 million belong to the image encoder, matching the 34M stated in the paper.
• Even when the input is 512×288, the model upscales the image to 1024×1024 for encoding. Consequently, most of the single detection time (67.0ms out of 69.4ms) is spent on the encoder. Sending a smaller phone screen resolution does not reduce this model's computation.
• Moving from FP32 to 16-bit made the encoder 3.1x faster. On the same GPU, the ball model (convolutional) was 1.5x faster (Part 2). This is because the large matrix multiplications in Transformers make better use of 16-bit dedicated computing units (Tensor Cores).
• Batch size 4 showed nearly identical time per image. This GPU is already saturated with a single image (Part 6).
• The 512×512 version was created by scaling position embeddings up and down, so accuracy was not measured(unmeasured). As shown in Table 5 of the paper, accuracy may drop if not retrained separately. Looking solely at execution time, running on one-fourth of the pixels of 1024 was 4.6x (FP32) to 3.1x (BF16) faster.

7. Limitations

• Computational volume is not elapsed time. The measurements in Sections 2 and 4 are examples of this. You should not select models based solely on computational volume or parameter counts in tables; you must measure them on the target device.
• What is lost by making reductions depends entirely on the task. A 1%p drop in MobileNet refers to ImageNet classification metrics. There is no guarantee of losing the same proportion when detecting a ball consisting of just a few pixels on a screen (unmeasured).
• Pruning ratios only indicate storage size. Speed gains are only achieved when structures are structurally removed or when dedicated kernels and hardware are available.

8. Connection to Our Project

• TT Score's ball model (BlurBall) belongs to the HRNet family but is already small, with 16–128 channels and 1.49M parameters. Despite this, the reason a single frame takes several to dozens of milliseconds at 512×288 is due to its architecture that preserves high resolution all the way through(the ball is only a few pixels wide, so resolution cannot be reduced). Therefore, rather than trying to make this model even smaller, we prioritized precision (Part 2) and input size (Part 5).
• Since finding the ping-pong table only needs to happen once at the start of a match, EfficientTAM's size is not an issue. As shown by the measurements in Section 6, if a single detection takes 30–70ms, the next step is "detection on the phone" (Part 7).
• "Recorded video teacher → Real-time student" distillation is ongoing research for TT Score. We plan to choose the student architecture using the measurement method from this article (measuring directly on the target device).

The next article explores how to run the model less frequently and over smaller areas while keeping it unchanged.

TT's Note: We reduced multiplications by 8x, but time only decreased by 1.6x. Sometimes reading data is slower than computing it. Trust the stopwatch (real measurements) over the scale (computational volume)!

References

• Howard, A. G. et al. (2017). MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications. arXiv. https://arxiv.org/abs/1704.04861
• Sandler, M. et al. (2018). MobileNetV2: Inverted Residuals and Linear Bottlenecks. CVPR 2018. https://arxiv.org/abs/1801.04381
• Tan, M., Le, Q. V. (2019). EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks. ICML 2019. https://arxiv.org/abs/1905.11946
• Ma, N. et al. (2018). ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design. ECCV 2018. https://arxiv.org/abs/1807.11164
• Han, S., Mao, H., Dally, W. J. (2016). Deep Compression. ICLR 2016. https://arxiv.org/abs/1510.00149
• Li, H. et al. (2017). Pruning Filters for Efficient ConvNets. ICLR 2017. https://arxiv.org/abs/1608.08710
• Xiong, Y. et al. (2024). Efficient Track Anything. arXiv. https://arxiv.org/abs/2411.18933
• Sample code and outputs: In this folder, code/04_small_models.py, code/04_small_models.out.txt