Tag: #long-context

← All posts

• The Training Recipe — From Pre-training to Post-training, and What Reports Write Down

  Comparing, exactly as written in the reports, the three stages and six context extensions of Llama 3, the three-stage pre-training of Qwen3, the learning-rate schedule and two-phase YaRN extension of DeepSeek-V3, and the

  2026-08-12 · 6 min read #ai-papers#model-internals#pretraining#training-recipe#data-mixture

• Positional Encoding — RoPE and the Price of Context Extension

  Showing through wavelength arithmetic how a single ropetheta value acts on context length, then comparing the 500000 of Llama 3, the ABF step in Qwen3, the YaRN settings of DeepSeek-V3 and Kimi K2, and the partial rotati

  2026-08-12 · 6 min read #ai-papers#model-internals#rope#positional-encoding#long-context

• RAG · Fine-Tuning · Long-Context — Which One for My Problem: What the Papers Actually Measured, and What No One Measured ♪ Listenable

  It's the most common question in LLM architecture, yet most answers are decision trees with no sources. This post answers only with what has been measured. That fine-tuning fails at injecting new knowledge has been measu

  2026-07-17 · 23 min read #rag#llm#fine-tuning#long-context#ai

• Did Context Engineering Replace Prompt Engineering — What's Measured and What Isn't ♪ Listenable

  "Prompt engineering is dead" appears in none of the primary sources that coined the term. Karpathy listed few-shot examples and task descriptions as components of context engineering, and Anthropic called it "a natural p

  2026-07-17 · 20 min read #llm#context-engineering#prompt-engineering#long-context#ai-agent

• Upcycling Pretrained Checkpoints into Long-Context Hybrids — What HyLo Proposes

  Hybrid sequence models (attention plus linear/SSM blocks) suit long context, but until now most had to be pretrained from scratch. The April 2026 preprint HyLo proposes a recipe to 'upcycle' an already-trained Transforme

  2026-07-11 · 5 min read #ai#llm#long-context#architecture#efficiency

• Understanding Positional Encoding — From Sine Waves to RoPE

  Starting from why Transformers need positional information, this article explains sinusoidal, learned, and relative positional encodings step by step, then RoPE and ALiBi. It connects length extrapolation and context ext

  2026-06-26 · 17 min read #llm#positional-encoding#rope#alibi#long-context

• The Evolution of Attention — MQA, GQA, FlashAttention, and Long Context

  We analyze the memory and compute cost of standard attention, then explain how MQA and GQA shrink the KV cache and how FlashAttention optimizes IO. We compare sliding-window and long-context techniques and trace how all

  2026-06-26 · 18 min read #llm#attention#flashattention#gqa#mqa

• Gemini 2.5 for Developers: A Practical Guide to Pro, Flash, and Flash-Lite

  A practical guide to Gemini 2.5 as of April 12, 2026, covering how to choose Pro, Flash, or Flash-Lite, how reasoning changes workflow design, and what teams should actually ship in production.

  2026-04-12 · 6 min read #google#gemini#gemini-2-5#coding#agentic-coding

• 1 Million Token Context Windows: Is RAG Becoming Obsolete?

  With Gemini 1.5 Pro's 1M token context and Claude's 200K, many developers ask: why bother with RAG? We compare real costs, latency, and quality to give you an honest answer.

  2026-03-18 · 6 min read #context-window#rag#llm#long-context#ai-development

• LLM Utilization Strategies in the 1M Context Window Era: A Practical Guide to Large-Scale Context Processing ♪ Listenable

  In March 2026, Anthropic announced GA of the 1M token context window for Claude Opus 4.6/Sonnet 4.6. Covers the paradigm shift from 128K-200K to 1M context, 5 practical patterns, RAG trade-offs, and cost optimization str

  2026-03-15 · 41 min read #ai-platform#llm#context-window#long-context#claude

• Ring Attention Paper Analysis: Implementing Infinite Context Window Training in Distributed Environments ♪ Listenable

  Analyzes the Ring Attention paper exploring methods to overcome context length limitations in distributed environments. Covers the connection with Blockwise Parallel Transformer, implementation details, performance bench

  2026-03-08 · 33 min read #ai-papers#ring-attention#distributed-training#long-context#transformer

• Complete Guide to LLM Long-Context Performance and KV Cache Optimization: From MQA to Ring Attention

  A comprehensive practical guide covering KV Cache fundamentals for LLM long-context processing, memory consumption analysis, optimization techniques including MQA, GQA, PagedAttention, sliding window attention, and Ring

  2026-03-07 · 17 min read #llm#kv-cache#long-context#multi-query-attention#grouped-query-attention