Tag: #training-stability
Writing on GPUs, LLMs, MLOps, Kubernetes — and mindset · 2 posts
Normalization and Activation — Keeping Training From Falling Apart
Confirming from config values why RMSNorm, pre-norm, and SwiGLU became the defaults, then walking through the newer devices that stop attention logits from exploding — the QK-Norm of Qwen3 and the QK-Clip of Kimi K2 — wi
2026-08-12 · 6 min read #ai-papers#model-internals#rmsnorm#swiglu#training-stabilityLessons from Published Training Runs — What Was Tried, and What Failed
Seven published large-scale training technical reports and logbooks, stripped down to just the failures and the responses to them, not the scoreboard. The statistics of 419 interruptions over 54 days from Llama 3 405B le
2026-08-02 · 13 min read #mlops#llm-training#case-study#training-stability#scaling