Tag: #LLM 사전학습
Kubernetes, DevOps and MLOps, written from things we ran and measured ourselves · 2 posts
AdamW vs Muon vs MUON+ Measured on a Small GPT
We trained a 34M-parameter GPT on 100M tokens of FineWeb-Edu, varying only the optimizer. Even after independently tuning the learning rate for each optimizer, matching the actual weight decay intensity, and aligning the
2026-10-31 · 17 min read #Muon#MUON+#AdamW#옵티마이저#LLM 사전학습Muon Commentary: An Optimizer that Orthogonalizes the Gradient Matrix
When reading 2026 LLM pretraining papers, Muon frequently appears in place of AdamW. Muon replaces the momentum matrix with a matrix whose singular values are all 1, evenly distributing updates that were clustered in a f
2026-10-31 · 15 min read #Muon#옵티마이저#AdamW#Newton-Schulz#MUON+