Muon Commentary: An Optimizer that Orthogonalizes the Gradient Matrix

Status: Draft (2026-10-31). Paper figures were transcribed directly from the arXiv original (the body text is from the HTML of the same version) along with table and figure numbers. The singular value figure (Section 3) was measured directly by the author on a laptop CPU. Training measurements are in Part 2. Writing Rules: Measured data is the default; unmeasured items are marked as 「Unmeasured」, and unverified items are marked as 「Unverified」.

Table of Contents

• 1. Optimizers Determine 「How Much and in Which Direction」
• 2. Viewed as a Matrix, Updates are Concentrated in a Few Directions
• 3. Orthogonalizing without SVD: Newton–Schulz 5 Times
• 4. Overall Algorithm and 「Where」 It Is Used
  • Size Correction Conventions Vary by Paper
• 5. Variants of 2025–2026
  • Moonlight: Weight Decay and Size Correction for Scaling Up
  • Preserving Data Efficiency at Large Batches (2505.02222)
  • MUON+: A Single Normalization After Orthogonalization (2602.21545)
  • Spectral Allocation: Different Steps for Different Directions (2608.25990)
  • Large-Scale Comparison: SOAP, Muon, and Beyond (2607.20548)
  • Applying Small Model Recipes to the Muon Family as Well
• 6. Conclusion and What We Want to Verify
• Raw Data and References

1. Optimizers Determine 「How Much and in Which Direction」

A training step is the process of looking at the gradient G and moving the weights W slightly. The simplest SGD is W ← W − η·G , moving a lot in directions where the gradient is large and a little where it is small. Adam adds to this by remembering 「how large the gradient for this coordinate has been so far」 for each coordinate and dividing by that magnitude. As a result, the steps for each individual coordinate become similar. This stability is the reason why LLM pretraining has used AdamW (Adam + decoupled weight decay) as its default for several years.

However, Adam treats weights as a bundle of numbers. Most transformer weights are matrices, and matrices have a structure called directionsrather than coordinates. Muon starts from here.

2. Viewed as a Matrix, Updates are Concentrated in a Few Directions

Any matrix can be expressed via Singular Value Decomposition (SVD) as G = U·Σ·Vᵀ . The columns of U and V are mutually orthogonal directions, and the diagonal elements of Σ (singular values) are the magnitudes in those directions. The gradient and momentum matrices of language models are known to often have singular values concentrated in a small number of entries (Keller Jordan's post on Muon describes this as 「nearly low rank」). Consequently, whether using SGD or Adam, actual updates move in only a few strong directions, while the remaining directions learn almost nothing.

Muon's prescription is simple. Use U·Vᵀ , where all singular values are changed to 1, as the update. It keeps the directions intact while equalizing the magnitudes. U·Vᵀ is the 「orthogonal matrix」 closest to the original matrix, which is why this operation is called orthogonalization.

3. Orthogonalizing without SVD: Newton–Schulz 5 Times

Performing SVD on large matrices at every step is slow. Muon approximates the same thing using only matrix multiplications. After dividing X by the Frobenius norm to make its singular values 1 or less, it runs the following iteration five times.

def newton_schulz5(G, steps=5, eps=1e-7):
    a, b, c = 3.4445, -4.7750, 2.0315      # Keller Jordan 이 고른 계수
    X = G.bfloat16()
    if G.size(0) > G.size(1):               # 세로로 긴 행렬은 눕혀서 계산
        X = X.T
    X = X / (X.norm() + eps)                # 특잇값을 1 이하로
    for _ in range(steps):
        A = X @ X.T
        B = b * A + c * A @ A
        X = a * X + B @ X                   # 특잇값 s 가 a·s + b·s³ + c·s⁵ 로 바뀐다
    if G.size(0) > G.size(1):
        X = X.T
    return X

A single iteration applies a polynomial a·s + b·s³ + c·s⁵ to each singular value (leaving the directions U and V unchanged). It is chosen to quickly grow s with small coefficients, so even small singular values near 0 rise close to 1 after five iterations. However, they do not converge to exactly 1 because the coefficients sacrifice a bit of accuracy for the sake of 「fast growth」.

I measured this directly. I created a 512×2048 matrix with singular values decreasing as 1, 1/2, 1/3, … (shaped like a gradient concentrated in a few directions), ran the above function in bf16, and measured the singular values at each iteration (Code: Raw Data section).

In the input, the top 10 out of 512 directions accounted for 93.7% of the energy. After five iterations, it became 2.5%, and all directions attained nearly equal magnitudes (0.68–1.21). It is quite close to the exact U·Vᵀ(1.95%). The cost is 15 matrix multiplications, and because it runs in bf16, it is inexpensive on GPUs, which is the practical core of Muon. The overhead measured in actual training is in Part 2 (5.4% throughput reduction, 1.5% when scaling batch size 4x).

4. Overall Algorithm and 「Where」 It Is Used

Following Keller Jordan's public implementation (linked in the blog muon.py), a single step looks like this.

1. Momentum: M ← β·M + (1−β)·G, β=0.95, using a Nesterov form (1−β)·G + β·M for the update.
2. Orthogonalization: O = NS5(위의 값).
3. Size Correction: O ← O · sqrt(max(1, 행/열)). This line is absent from the blog body and appears only in the code.
4. Update: W ← W − η·O(default η=0.02).

What is important is the scope of application. Muon is used exclusively for 2D matrices in hidden layers (Q, K, V, O in attention, and the two matrices in MLPs). Embeddings, output layers (lm_head), normalization layer gains, and biases are left to AdamW. This is because embeddings are tables (lookups) that receive gradients for only a few rows at a time, making the notion of 「matrix directions」 inapplicable, and 1D parameters have no matrices to orthogonalize. If your implementation merges Q, K, and V into a single matrix, it is recommended to split them into three and orthogonalize them separately.

Size Correction Conventions Vary by Paper

For orthogonal matrices, the size (RMS) of a single element is determined by the matrix shape. Consequently, the line determining 「how much to scale the update」 varies across papers, and the same 「learning rate of 0.02」 ends up having different meanings.

Moonlight's (arXiv 2502.16982) 0.2·sqrt(max) is a device to adjust the RMS of updates to AdamW's typical value of 0.2, allowing learning rates and weight decays tuned for AdamW to be carried over directly. For a 512×512 matrix, Keller's convention of 0.02 corresponds to approximately 4.4e-3 under Moonlight's convention (calculation). This is the most frequent point of confusion when porting learning rates from papers.

5. Variants of 2025–2026

Moonlight: Weight Decay and Size Correction for Scaling Up

Moonlight showed that using Muon on large models requires two things. Without adding decoupled weight decay (W ← W − η(O + λW)), weight and output sizes keep growing, and the fact that update sizes vary by matrix shape must be adjusted via 0.2·sqrt(max). They report achieving the same performance at roughly 52% of AdamW's FLOPs under compute-optimal conditions (Figure 1a, Table 3).

Preserving Data Efficiency at Large Batch Sizes (2505.02222)

Essential AI's 「Practical Efficiency of Muon for Pretraining」 looks at scaling up batch sizes. Increasing batch sizes reduces the number of steps and speeds things up, but beyond a certain point (the critical batch size), more tokens are required to reach the same loss. This paper plotted the ratio of tokens used by AdamW and Muon to reach the same loss across batch sizes, noting that Muon uses 10–15% fewer tokens and that this advantage is maintained or grows even as batch sizes increase (Figures 2 and 3). The original phrasing is "retaining data efficiency at large batch sizes, far beyond the so-called critical batch size". Rather than relying on point estimates that 「the critical batch size is larger」, the basis lies in the token ratio not decreasing. We will re-test this claim on a small scale in Part 2.

MUON+: A Single Normalization After Orthogonalization (2602.21545)

While orthogonalization would seem to equalize row and column sizes, the paper points out that the actually used 5-iteration process can instead exacerbate the imbalance of row and column norms (Section 2.1, Figures 2 and 3). This is rooted in the same issue as 5 iterations not being an exact U·Vᵀ (the fact that singular values were spread out from 0.68 to 1.21 in the table above). The authors analyzed that this imbalance reduces the maximum stable step size. MUON+ adds an extra step of dividing by the column norms of the orthogonalization result, followed by dividing by the row norms. It adds no states and takes only a few lines of code. Across GPT and LLaMA models from 60M to 7B, it achieved lower validation PPL than Muon (LLaMA-60M 25.75 → 25.25, Table 2) and reported reducing steps to the same loss by 22.4–37.1% (Table 5). It is also worth remembering that for LLaMA-60M, which is close in size to our experimental model, the gain was as small as roughly 0.5 PPL.

Spectral Allocation: Different Steps for Different Directions (2608.25990)

Muon assigns the same step to all directions. This paper directly measured the 「step that most reduces loss on unseen batches」 for each singular direction of the momentum matrix during training. The result is that the topmost directions hit the Edge of Stability, whereas the remaining directions can tolerate several times larger steps. Consequently, they proposed SAMuon, which keeps the top directions at the Muon size while scaling the rest by γ, and reported using 13–24% fewer tokens to reach the same loss (Table 7). From this perspective, the order 「Muon > Adam > SGD」 is also explained. SGD concentrates steps on strong directions, Adam softens this only at the coordinate level, and Muon equalizes it at the direction level. In Table 5 of this paper, the loss difference between AdamW and Muon widened as the batch size increased, being 0.048 for a sequence batch of 1024 and 0.110 for 4096.

Large-Scale Comparison: SOAP, Muon, and Beyond (2607.20548)

This paper by NVIDIA trained 8B dense and 30B/72B MoE models with 1–3T tokens, comparing AdamW, Muon, and SOAP. Muon and SOAP consistently outperformed AdamW and held up even when scaling the global batch size up to 100M tokens, whereas AdamW degraded (Abstract, Section 5.3). KL-SOAP, which fixes the large-batch instability of SOAP, edged out Muon "consistently but by a narrow margin" (Figures 7 and 8). However, the authors clarify that their fair comparison did not tune the learning rate separately for each optimizer, but rather matched and transferred the update RMS. The original text also notes that AdamW actually had a higher mathematical average for the 8B hybrid model (Table 5, 84.04 vs. 80.79).

Small Model Recipes Also Adopt the Muon Family

Two small-model pretraining recipes released in 2026 also use the Muon family for the optimizer slot. IMU-1 (2602.02522, 430M) used NorMuon with Cautious Weight Decay and $\mu$P, while Puro-2B (2608.27370) used MuonH. In both cases, embeddings, output layers, and 1D parameters use AdamW. This branch will be revisited in Part 4.

6. Conclusion and Questions to Verify

• Muon modifies the hidden matrix update to U·Vᵀ to give even step sizes across all directions. The computation consists of five matrix multiplications.
• Learning rate numbers only become meaningful when the usage (hidden 2D matrices only) and scale correction protocol are specified together.
• The trends in 2026 are three-fold: "refining the orthogonalization results" (MUON+), "varying step sizes per direction" (Spectral Allocation), and "fair comparisons at scale and large batch sizes" (NVIDIA, Essential AI).

The papers reached these conclusions at hundreds of millions to hundreds of billions of parameters. Will the same thing happen with a 34-million parameter model and 100 million tokens that can run on a single 8GB laptop GPU? How much of a difference remains if we tune the learning rate separately for each optimizer for a fair comparison? Will the gap widen at large batch sizes? Part 2measures this directly.

Raw Data and References

• Singular value measurement code: ns_spectrum.py(the experiment folder for this series). Laptop CPU, PyTorch 2.15, seed 0. The same NS function is used by the training code in Part 2.
• Keller Jordan, "Muon: An optimizer for hidden layers in neural networks" (blog post, 2024) and public implementation muon.py.
• Liu et al., "Muon is Scalable for LLM Training", arXiv 2502.16982.
• Essential AI, "Practical Efficiency of Muon for Pretraining", arXiv 2505.02222.
• Zhang et al., "MUON+", arXiv 2602.21545.
• Wu et al., "Spectral Allocation", arXiv 2608.25990.
• Khona et al., "SOAP, Muon, and Beyond", arXiv 2607.20548.