Tag: #2026 LLM 학습 논문 읽기

← All posts

• Trends in LLM Training Research for 2026: Optimizers, Data, Distillation, and RL

  We summarize LLM training papers from the second half of 2026 by category. In pretraining, the Muon family is becoming the default optimizer, a scaling law was published showing that web data's AI text proportion exceeds

  2026-10-31 · 14 min read #LLM 학습#사전학습#데이터 큐레이션#지식 증류#강화학습

• Does RL Selectively Filter: GRPO and pass@k Measured on a 0.5B Model

  Does RL post-training create new capabilities, or does it select answers that the base model occasionally generated? We ran GRPO directly on a 0.5B instruction model using three-digit multiplication arithmetic and measur

  2026-10-31 · 24 min read #강화학습#GRPO#RLVR#pass@k#LoRA

• AdamW vs Muon vs MUON+ Measured on a Small GPT

  We trained a 34M-parameter GPT on 100M tokens of FineWeb-Edu, varying only the optimizer. Even after independently tuning the learning rate for each optimizer, matching the actual weight decay intensity, and aligning the

  2026-10-31 · 17 min read #Muon#MUON+#AdamW#옵티마이저#LLM 사전학습

• Muon Commentary: An Optimizer that Orthogonalizes the Gradient Matrix

  When reading 2026 LLM pretraining papers, Muon frequently appears in place of AdamW. Muon replaces the momentum matrix with a matrix whose singular values are all 1, evenly distributing updates that were clustered in a f

  2026-10-31 · 15 min read #Muon#옵티마이저#AdamW#Newton-Schulz#MUON+