Tag: #ring-attention
Writing on GPUs, LLMs, MLOps, Kubernetes — and mindset · 3 posts
Foundation Model Architectures 2026 — Beyond the Transformer / Mamba 2 / Hyena / RWKV / RetNet / Griffin / Jamba / xLSTM / TTT / DiT / MoE / Flash Attention 3 Deep Dive ♪ Listenable
In 2026 the foundation-model world is no longer Transformer-only. Vaswani 2017 "Attention is All You Need" remains the standard, but next to it stand state-space models (Mamba, Mamba 2), the linear-RNN renaissance (RWKV,
2026-05-16 · 22 min read #foundation-models#transformer#attention-is-all-you-need#vaswani#mambaRing Attention Paper Analysis: Implementing Infinite Context Window Training in Distributed Environments ♪ Listenable
Analyzes the Ring Attention paper exploring methods to overcome context length limitations in distributed environments. Covers the connection with Blockwise Parallel Transformer, implementation details, performance bench
2026-03-08 · 33 min read #ai-papers#ring-attention#distributed-training#long-context#transformerComplete Guide to LLM Long-Context Performance and KV Cache Optimization: From MQA to Ring Attention
A comprehensive practical guide covering KV Cache fundamentals for LLM long-context processing, memory consumption analysis, optimization techniques including MQA, GQA, PagedAttention, sliding window attention, and Ring
2026-03-07 · 17 min read #llm#kv-cache#long-context#multi-query-attention#grouped-query-attention