Tag: #attention-mechanism
Writing on GPUs, LLMs, MLOps, Kubernetes — and mindset · 2 posts
KV Cache Optimization Deep Dive: GQA, MLA, and MHA Attention Mechanisms with Memory Efficiency Strategies
From Transformer Self-Attention KV Cache fundamentals through MHA, MQA, GQA (Llama 2/3), and MLA (DeepSeek-V2/V3) mechanism memory analysis and comparison, KV Cache compression techniques (quantization, eviction policies
2026-03-11 · 17 min read #ai-papers#kv-cache#attention-mechanism#gqa#mlaFlashAttention Paper Analysis: Revolutionizing Transformer Training and Inference with IO-Aware Exact Attention ♪ Listenable
An in-depth analysis of the FlashAttention series (v1 through v3). Covers the tiling strategy of the IO-aware algorithm, GPU SRAM/HBM memory hierarchy utilization, backward pass recomputation, parallelism improvements in
2026-03-09 · 22 min read #ai-papers#flash-attention#transformer#gpu-optimization#attention-mechanism