Skip to main content

Learn Learning paths Courses

Transformers — Compute Attention By Hand

Confirm with numbers why you divide by √d

고급 · Lessons 31 · Lab 10

Curriculum

Inside Attention

Attention Is a Weighted Average reading
Attention Without a Library lab
The Current Model Landscape and the Encoders reading
Transformer Check quiz

Tokenization and Vocabulary

The Same Sentence, a Different Token Count reading
Build a Tokenizer by Hand lab
Quiz: Tokenization and Vocabulary quiz

Rotary Position Embedding (RoPE)

Rotate the Position Instead of Adding It reading
Put Position In by Rotating lab
Quiz: Rotary Position Embedding quiz

Temperature, top-k and top-p

Context Length and Cost

What a KV Cache Reduces

Embeddings, Weight Tying, and Logits

How Integer Quantization Hits Accuracy

What Changes When You Fold Into int8 and Back reading
Doing int8 Arithmetic by Hand lab
Quiz: Integer Quantization Error quiz

MHA, MQA, and GQA

Keep the Query Heads, Shrink Only the Key/Value Heads reading
Shrink Only the Key/Value Heads lab
Quiz: MHA, MQA, and GQA quiz

Cross Entropy and Perplexity

Reference docs