Tag: #flash-attention
Writing on GPUs, LLMs, MLOps, Kubernetes — and mindset · 7 posts
7 posts are tagged #flash-attention; the most recent was published on 2026-05-16.
Often tagged together: #2026-03 3 #ai-papers 3 #transformer 3 #gpu 2 #model-serving 2
Most read in the last 90 days:
LLM Landmark Papers Roundup 2026 - Transformer / Scaling Laws / Flash Attention / Mamba / DeepSeek-R1 / Titans Deep Dive ♪ Listenable
From the 2017 Attention Is All You Need paper to 2026 Titans and DeepSeek-R1, a thematic roundup of the 50+ landmark papers that built the LLM era. Transformer, BERT, GPT 1-3, Scaling Laws, Chinchilla, InstructGPT, PaLM,
2026-05-16 · 21 min read #llm-papers#transformer#scaling-laws#flash-attention#mambaComplete LLM Serving Optimization Guide: KV Cache, PagedAttention, and Quantization ♪ Listenable
A complete engineering deep dive into LLM serving optimization. From the KV cache memory crisis to the virtual-memory revolution of PagedAttention, continuous batching, speculative decoding, quantization techniques, and
2026-03-18 · 30 min read #model-serving#kv-cache#paged-attention#vllm#quantizationHow Matrices Fly on GPU: Complete Deep Dive from GEMM to FlashAttention
Matrix multiplication accounts for over 80% of deep learning compute. This post dissects every layer of optimization from naive O(n³) code to cache blocking, cuBLAS, Tensor Cores, and the IO-aware revolution of FlashAtte
2026-03-18 · 15 min read #gemm#matrix-multiplication#flash-attention#gpu#parallel-computingLLM Pretraining & Scaling Laws: From Chinchilla to Flash Attention and MoE
A complete guide to LLM pretraining: Chinchilla scaling laws, Common Crawl data pipelines, Flash Attention 2, GQA, MoE architectures, and the latest pretraining recipes from DeepSeek-V3, Llama 3.1, and phi-4.
2026-03-17 · 14 min read #llmpretraining#scalinglaws#chinchilla#flash-attention#mixtralmoeAdvanced CUDA GPU Programming: Warp Optimization, Tensor Cores, and Triton Kernels
A comprehensive deep dive into CUDA memory hierarchy, Warp optimization, Tensor Core WMMA API, Flash Attention implementation, and Triton custom kernel authoring for accelerating AI model training.
2026-03-17 · 19 min read #cuda#gpuprogramming#tensorcore#triton#flash-attentionFlashAttention Paper Analysis: Revolutionizing Transformer Training and Inference with IO-Aware Exact Attention ♪ Listenable
An in-depth analysis of the FlashAttention series (v1 through v3). Covers the tiling strategy of the IO-aware algorithm, GPU SRAM/HBM memory hierarchy utilization, backward pass recomputation, parallelism improvements in
2026-03-09 · 22 min read #ai-papers#flash-attention#transformer#gpu-optimization#attention-mechanismFlashAttention: Optimizing Attention Through GPU Memory Hierarchy
A review of the FlashAttention paper with an in-depth analysis of IO-aware attention optimization leveraging the GPU HBM/SRAM memory hierarchy.
2026-03-01 · 18 min read #ai-papers#flash-attention#gpu#optimization#transformer