Tag: #DeiT
Kubernetes, DevOps and MLOps, written from things we ran and measured ourselves · 2 posts
Knowledge Distillation: Transferring What a Large Model Knows to a Small Model — Plus "Recorded Teacher → Real-Time Student"
Knowledge distillation is a training method where the outputs of a large model (the teacher) are used as the targets for a small model (the student). We examine Hinton et al.'s (2015) idea that teacher probabilities soft
2026-10-17 · 17 min read #비전 모델#지식 증류#온도#DeiT#Noisy StudentVision Transformer: Reading an Image Like a Sentence of 16×16 Patches, and Why It Needs So Much Data
We examine the principles of ViT (2021), which reads images using the Transformer originally designed for reading sentences. It cuts an image into 16×16 patches, lays them out like words, and uses self-attention to exami
2026-10-17 · 13 min read #비전 모델#ViT#트랜스포머#셀프 어텐션#위치 임베딩