Knowledge Distillation: Transferring What a Large Model Knows to a Small Model — Plus "Recorded Teacher → Real-Time Student"

Status: Draft (2026-10-13). Paper numbers were verified by opening the original texts and checking their conditions. The hands-on portion is a toy example on a laptop CPU (code/04_distill_toy.py, 2–3 minutes). The "Recorded Teacher → Real-Time Student" distillation for TT Score is at the planning stageand has not yet been implemented on actual table tennis data (ongoing research, unmeasured). Part 4 of the series "Reading Vision Models 2." Part 2's DINO was already "self-distillation": Self-Supervised Learning 1.

Table of Contents

• 1. Intuition: A Teacher That Gives Only the Answer vs. a Teacher That Explains Why You Might Get Confused
• 2. How to Train
  • Softening with Temperature
  • Transfer Set: Labels Are Not Required
  • What to Match: Outputs, Intermediate Features, Tokens
• 3. Diagram: Our Plan
• 4. Hands-On
  • A. A Miniature Version of Classical Distillation
  • B. Recorded Teacher → Real-Time Student
• 5. Paper Statistics
• 6. Limitations
• 7. Connection to Our Project
• References

1. Intuition: A Teacher That Gives Only the Answer vs. a Teacher That Explains Why You Might Get Confused

If you show a handwritten digit "2" and say only "The answer is 2," the student learns to treat 2 and the other nine digits as equally different. However, when you show the same image to a well-trained large model, it responds like this: "Probability 0.97 of being 2, probability 0.02 of being 3, probability 0.009 of being 7, and nearly 0 for the rest." There is information in these small numbers. This 2 is slightly similar to 3 or 7, and not at all similar to 4. This is information not found in the ground-truth label.

Hinton et al. (2015) called this "dark knowledge" and proposed using the probability outputs of a large model (the teacher) as targets for a small model (the student). This is knowledge distillation . The student is much smaller and faster than the teacher, inheriting the "map of similarities" built by the teacher in return.

2. How to Train

Softening with Temperature

The probabilities of a well-trained teacher are usually close to 0 or 1. Values like 0.009 in the example above have no presence in the loss. Therefore, logits are divided by a temperature T before being fed into the softmax function (the same knob as τ in Part 1, but used with a larger value in the opposite direction).

p_교사(T) = softmax(z_교사 / T),   p_학생(T) = softmax(z_학생 / T)
L = (1 − α) · CE(정답, p_학생(1))  +  α · T² · KL(p_교사(T) ‖ p_학생(T))

• When T is large, small probabilities revive, revealing similarity information in the loss.
• Why Multiply by T²: The magnitude of the gradients from the softened component decreases roughly by a factor of 1/T². We scale it back up so that the balance between the two terms is maintained even if T is changed (from the paper).
• Mixing in the correct answer term with a small weight is usually better, as teachers can sometimes be wrong.
• During testing, the student reverts to T=1.

Transfer Set: Labels Are Not Required

The distillation term does not require ground-truth labels. If you feed any input into the teacher, its output serves directly as the target. Therefore, the data used to train the student (the transfer set) can be unlabeled data. This property is the core of our plan at the end of this post.

What to Match: Outputs, Intermediate Features, Tokens

• Outputs (Logits): The basic form described above.
• Intermediate Features (FitNets, 2015): First train the student's intermediate layers to mimic the teacher's intermediate features (hints), and then distill the entire network. This is an approach to solve the difficulty of training thin and deep students.
• Distillation Token (DeiT, 2021): Add one more "distillation token" in addition to the classification token in a ViT student, and have this token learn to match the teacher's answer.

3. Diagram: Our Plan

TT Score judgment has two use cases: real-time judgment, which displays scores on the screen during a match, and recorded analysis, which re-analyzes videos after the match ends. The two have different constraints.

The recorded version will likely be more accurate because it can make judgments by seeing where the ball goes even after a bounce. If that teacher labels hundreds of unlabeled recordings, the real-time student can learn from far more examples than rallies manually labeled by humans. This plan is noted in the research notes as "Two Tracks (Best Real-Time, Best Recorded) and Distillation," and has not yet been executed (ongoing research).

4. Hands-On

code/04_distill_toy.py was run on a laptop CPU (outputting code/04_distill_toy.out.txt).

A. A Miniature Version of Classical Distillation

This is a 10-class classification task with mutually similar pairs (0 and 5, 1 and 6, etc.). The teacher (278,000 parameters) was trained with 3,000 labels, achieving a test accuracy of 0.856. The student has 754 parameters (0.27% of the teacher).

"Dark knowledge" is revealed when examining the teacher's outputs across temperatures (test image where the true label is 7).

T=1 → [0.    0.    0.    0.    0.    0.    0.    1.    0.    0.   ]
T=4 → [0.005 0.004 0.006 0.    0.001 0.    0.008 0.962 0.002 0.012]

At T=1, only the correct answer is visible, but when softened with T=4, information emerges showing that class 7 is slightly similar to classes 6 and 9, and hardly similar to classes 3 and 5.

Using the teacher's output directly at T=1 yielded almost no effect, while softening to T=4 improved accuracy by about 2 percentage points. Furthermore, without using a single human label, without using any human labels a student trained solely on outputs generated by the teacher for 3,000 unlabeled data points reached 0.854, nearly matching the teacher (0.856)—even though the student's size is 0.27% of the teacher's.

We also replicated one of Hinton et al.'s famous experiments: removing all samples appearing to be the digit 3 from the transfer set. The student never sees the digit 3 as a "correct answer."

전이 세트에서 3번을 뺀 학생: 3번 재현율 0.066, 전체 0.774
  3번 출력의 치우침(bias)만 +8 올리면: 3번 재현율 0.905, 전체 0.845

Although it can barely recognize 3, boosting just the bias for 3 allows it to get 90% correct. The student learned what the digit 3 looks like from the "similarity to 3" mixed into the outputs of other numbers. Note that the bias value was chosen for demonstration by looking at the test set. The original paper's MNIST experiment followed the same direction (table below).

B. Recorded Teacher → Real-Time Student

This is the problem of finding bounce frames from ball height sequences (60fps, positional noise, 10% dropped frames, slow jitter simulating camera parallax). We also mixed in shots where the ball hits the racket and changes direction, making it impossible to be certain whether it's a bounce based solely on the deflection shape.

• Teacher: Observes 8 frames before and after the judgment frame (recorded).
• Student: Observes the past 16 frames plus either 0 or 3 lookahead frames (50ms) (real-time).
• 20 rallies with human labels, 300 unlabeled rallies, 40 validation rallies, and 200 test rallies. Thresholds are chosen on the validation rallies, and scoring is based on F1 matching bounces within ±2 frames.
• Labels are 1 for bounce frames and 0.5 for immediately adjacent frames. Since bounce frames make up only 2.9%, positive instances were weighted.

The 0-lookahead student improved from 0.860 to 0.886 using the teacher's labels, which is nearly identical to the performance when humans directly labeled the same 300 rallies (0.892). The teacher effectively replaced 300 human-labeled rallies. The 3-lookahead student also reached 0.886, but started from a higher baseline, so the gain was smaller.

The gap with the teacher (0.923) was not closed by distillation. No teacher, no matter how good, can transfer information contained in the future that the student cannot see. Distillation bridges the "gap caused by insufficient data," not the "gap caused by insufficient input." This distinction determines what we can and cannot expect from our plan.

5. Paper Statistics

A few things stand out.

• It is not uncommon for the student to outperform the teacher(FitNets, DWPose, Noisy Student). This means that the teacher's output is sometimes an answer "easier to learn" than the ground-truth label.
• Distillation takes longer than expected. Beyer et al. viewed distillation as "tracing the teacher's function as is," concluding that the teacher and student must be shown the same inputs and trained for a very long time.
• Unlabeled data is the real strength. Noisy Student and our Toy B tell the same story.

6. Limitations

• The teacher's mistakes are also transferred. Labels incorrectly marked as "bound" by the teacher become absolute truth for the student. This is also why Noisy Student filtered out labels with low confidence (excluding B0 confidence under 0.3).
• Information that cannot be seen cannot be transferred. This is the lesson from Experiment B above. We should not expect a real-time student to have all of the recorded teacher's "future-seeing judgments".
• If the domain differs, the teacher is wrong. If the teacher is trained only on a public dataset (side-fixed camera), even the teacher might be wrong on our videos shot at an angle with a mobile phone (estimation).

7. Connection to Our Project

• The real-time student already exists. The learning-based bound discriminator is a 1D convolutional model looking at the past 16 frames of the ball trajectory (lookahead 0), with about 18,000 parameters. It filters out bounds generated by the rule engine that have a low bound probability according to this model. As a result of finalizing a board trained with 30fps copies mixed across 5 seeds, when scoring 7 OpenTTGames test matches like recorded footage, the rally winner accuracy under the sum of 9 degradation conditions was +1.4%p(95% interval +0.3…+2.5) and did not drop at 30fps. When putting the same model directly into the actual deployment streaming pipeline, under 11 degradation conditions and 627 rallies, it was +4.9%p(+2.9…+7.2), with incorrect judgments without verification dropping from 50 to 34 cases. The latter has been adopted and is undergoing operational deployment (as of Research Director 2026-10-13, ongoing research — the director is still outside the repository). Initial results collapsed at 30fps, so we changed the training data and remeasured. We also note that since the same 57 rallies are recounted across 11 conditions, the interval is optimistic, and that the rule engine was refined while viewing the test matches, making the baseline optimistic.
• Teacher candidates. On the recorded footage side, we plan to test a clip-level precise event model (Part 10), an occlusion-robust ball tracker (Part 9), and a method that makes judgments after lifting to 3D (Part 11). We haven't run these in this round yet.
• Sound-side precedents. EfficientAT, examined as a candidate in research on hearing bounds by sound, is a model created through "large Transformer → small CNN" distillation (TT Score Model Anatomy Part 6).
• Pose-side precedents. DWPose is a model that scales up the RTMPose series used in TT Score Coaching through distillation (Part 8).

TT's Word. If a coach who watched the match to the end marks "this was a bounce" across hundreds of rallies, even a student watching only in real-time learns quite well. Still, since it cannot teach the future that has not arrived yet, the student remains slightly less skilled than the coach.

References

• Hinton, Vinyals, Dean, Distilling the Knowledge in a Neural Network, NIPS 2014 Deep Learning Workshop(arXiv 2015). https://arxiv.org/abs/1503.02531
• Romero et al., FitNets: Hints for Thin Deep Nets, ICLR 2015. https://arxiv.org/abs/1412.6550
• Touvron et al., Training data-efficient image transformers & distillation through attention(DeiT), ICML 2021. https://arxiv.org/abs/2012.12877
• Beyer et al., Knowledge distillation: A good teacher is patient and consistent, CVPR 2022. https://arxiv.org/abs/2106.05237
• Xie et al., Self-training with Noisy Student improves ImageNet classification, CVPR 2020. https://arxiv.org/abs/1911.04252
• Schmid et al., Efficient Large-Scale Audio Tagging via Transformer-to-CNN Knowledge Distillation(EfficientAT), ICASSP 2023. https://arxiv.org/abs/2211.04772
• Yang et al., Effective Whole-body Pose Estimation with Two-stages Distillation(DWPose), ICCV 2023 Workshop. https://arxiv.org/abs/2307.15880
• Oquab et al., DINOv2, TMLR 2024. https://arxiv.org/abs/2304.07193
• Example code and output: code/04_distill_toy.py, code/04_distill_toy.out.txt