Tag: #attention
Writing on GPUs, LLMs, MLOps, Kubernetes — and mindset · 9 posts
9 posts are tagged #attention; the most recent was published on 2026-08-14.
Often tagged together: #llm 4 #kv-cache 3 #transformer 3 #ai 2 #ai-papers 2
Most read in the last 90 days:
The Hybrid Attention in Qwen3.8-27B — Only 16 of 64 Layers Grow a KV Cache ♪ Listenable
The reason a 27B model can carry a 262K-token context on a laptop is not its parameter count but its layer layout. Qwen3.8-27B places 48 of its 64 layers as linear attention (Gated DeltaNet) and only 16 as ordinary atten
2026-08-14 · 8 min read #llm#inference#quantization#local-llm#attentionAttention Variants — From MHA to MLA, and How the KV Cache Shrinks
Comparing MHA, MQA, GQA, and MLA using real config values. Working out with formulas and numbers how the grouped-query attention in Qwen3, Mixtral, and GLM-4.5 and the latent attention in DeepSeek-V3 and Kimi K2 reduce p
2026-08-12 · 6 min read #ai-papers#model-internals#attention#gqa#mlaOptimizing Long-Context Inference with Hybrid SWA — What Xiaomi's MiMo v2.5 Actually Does
Hybrid SWA interleaves sliding-window attention with full attention across layers to cut both KV-cache memory and compute for long-context inference — now a mainstream design. Grounded in Xiaomi's published MiMo v2.5 inf
2026-07-11 · 7 min read #ai#llm#inference#attention#optimizationReading Deeply, Thinking Slowly — The Craft of Reflection
In an age of endless skimming, we are slowly losing the habit of reading deeply. This essay looks calmly at the difference between shallow and deep reading, what the internet did to our attention, the arts of active and
2026-07-05 · 17 min read #learning#reading#reflection#attentionThe Truth About Dopamine Detox — Focus Myths and the Science ♪ Listenable
Dopamine is not a pleasure chemical but a signal of motivation and prediction. This piece examines the scientific basis of the trendy dopamine detox and, evidence in hand, cheerfully explores what really helps focus in a
2026-07-01 · 28 min read #culture#dopamine#attention#focus#neuroscienceDissecting the Transformer — From Attention to KV Cache
A from-scratch breakdown of the Transformer: self-attention, multi-head, positional encoding, the FFN, and residual connections with normalization. It connects tensor shapes and parameter counts, causal masking, encoder/
2026-06-26 · 16 min read #llm#transformer#attention#positional-encoding#kv-cacheThe Evolution of Attention — MQA, GQA, FlashAttention, and Long Context
We analyze the memory and compute cost of standard attention, then explain how MQA and GQA shrink the KV cache and how FlashAttention optimizes IO. We compare sliding-window and long-context techniques and trace how all
2026-06-26 · 18 min read #llm#attention#flashattention#gqa#mqaTransformer Architecture Complete Analysis: From Attention to Modern LLMs ♪ Listenable
A complete guide to understanding Transformer architecture from scratch. Covers Self-Attention, Multi-Head Attention, Positional Encoding, and Encoder-Decoder structure through Flash Attention, RoPE, and GQA with formula
2026-03-17 · 22 min read #transformer#attention#deep-learning#nlp#aiAttention Is All You Need - A Complete Analysis of the Transformer Paper
A thorough analysis of the "Attention Is All You Need" paper, the foundation of the Transformer architecture, breaking down its core mechanisms including Self-Attention, Multi-Head Attention, and Positional Encoding one
2026-03-01 · 18 min read #ai-papers#transformer#deep-learning#attention