Tag: #positional-encoding

← All posts

• Positional Encoding — RoPE and the Price of Context Extension

  Showing through wavelength arithmetic how a single ropetheta value acts on context length, then comparing the 500000 of Llama 3, the ABF step in Qwen3, the YaRN settings of DeepSeek-V3 and Kimi K2, and the partial rotati

  2026-08-12 · 6 min read #ai-papers#model-internals#rope#positional-encoding#long-context

• Understanding Positional Encoding — From Sine Waves to RoPE

  Starting from why Transformers need positional information, this article explains sinusoidal, learned, and relative positional encodings step by step, then RoPE and ALiBi. It connects length extrapolation and context ext

  2026-06-26 · 17 min read #llm#positional-encoding#rope#alibi#long-context

• Dissecting the Transformer — From Attention to KV Cache

  A from-scratch breakdown of the Transformer: self-attention, multi-head, positional encoding, the FFN, and residual connections with normalization. It connects tensor shapes and parameter counts, causal masking, encoder/

  2026-06-26 · 16 min read #llm#transformer#attention#positional-encoding#kv-cache

• Complete Guide to LLM Context Window Extension: From RoPE, ALiBi, and YaRN to Ring Attention

  An in-depth analysis of techniques for extending LLM context windows from 512 to 2M tokens. Covers the mathematical principles of RoPE through NTK-aware scaling, YaRN, and Ring Attention with practical code examples.

  2026-03-03 · 9 min read #llm#context-window#rope#yarn#positional-encoding