AdamW vs Muon vs MUON+ Measured on a Small GPT

Status: Draft (2026-10-31), measured. All training was run directly by the author on a single 8GB laptop GPU. The conditions, learning rate candidates, and decision criteria were written down before the first run (preregistration), and the execution proceeded automatically according to those rules. The paper figures are the original verified copies as in Part 1. Writing rules: Measured data is the default; unmeasured items are marked as "Unmeasured", and unverified items are marked as "Unverified".

Table of Contents

• 1. What We Are Verifying
• 2. Experimental Design
  • Model and Data
  • Things Held Constant
  • Three Optimizers
  • Learning Rate Sweep and Main Run (Preset Rules)
  • Environment
• 3. Learning Rate Sweep: Different Optimizers Have Different "Good Neighborhoods"
• 4. Main Run: 100M Tokens, 3 Seeds
  • How Many Tokens Are Required for the Same Loss
  • Time and Memory
• 5. Large Batch: The Gap Widened
• 6. Post-Hoc Checks: Three Questions Arising from the Results
• 7. Interpretation
• 8. Limitations
• Raw Data

1. What We Are Verifying

Part 1We distilled the claims of the papers seen in into three questions.

1. When changing only the optimizer under the same model, data, and tokens, even after tuning the learning rate separately for each optimizer , does Muon reach a lower loss than AdamW?
2. Does MUON+'s "one step of normalization after orthogonalization" make a difference even at this scale?
3. Does increasing the batch size widen the gap between AdamW and Muon (as claimed by 2505.02222 and 2608.25990)?

The scale is much smaller than the papers: a 34M-parameter model and 100M tokens. Therefore, the conclusion of this post is "this is how it behaved at this scale" rather than verifying or refuting the papers.

2. Experimental Design

Model and Data

• Model: A custom-written decoder-only transformer (nanoGPT family). Dimension 512, 8 layers, 8 heads, context 512, RMSNorm (pre-norm), RoPE, GELU MLP (4x), no bias, separate embedding and output layers. Total 33.56M parameters, of which 25.17M are in 2D hidden layer matrices.
• Data: Approximately the first 1 billion characters from the first chunk of the public FineWeb-Edu (ODC-By 1.0) sample. A byte BPE vocabulary of 8,192 was trained directly on this. 260M training tokens (1 token ≈ 3.9 characters), and 520k validation tokens out of 4,000 documents not overlapping with the training set.
• Reason for reducing vocabulary to 8k: Using the GPT-2 vocabulary (50k) would make the embedding (25.7M) larger than the hidden matrices in a model of this size. What we want to compare is the optimizer for the hidden matrices.
• Korean was not mixed in. In an 8k vocabulary built from English text, Korean is split at the byte level, and the mixing ratio would become yet another variable.

Things Held Constant

Data order (fixed permutation, all runs see the same sequences in the same order), token budget, batch size, bf16 automatic mixed precision, torch.compile, gradient norm clipping 1.0, learning rate schedule (WSD: initial 2% linear warmup, constant phase, final 20% linear decay to 0), evaluation set. Seeds change initialization only.

Three Optimizers

Two aspects were intentionally aligned. First, the auxiliary AdamW learning rate for the Muon run was fixed to the optimal value of the AdamW run, ensuring that the difference between the two runs is solely the "hidden matrix optimizer". Second, the weight decay rate per step (learning rate × λ) was made equal to AdamW. Muon's learning rate convention (around 0.02) differs from AdamW's (around 1e-3) by about twentyfold, so leaving λ identical at 0.1 results in a twentyfold difference in actual decay (Part 1, Section 4).

MUON+ differs from the paper's pseudocode in two places. The paper does not use Nesterov and uses sqrt(rows/cols) for size correction, whereas here it is kept identical to Muon with just one additional normalization step. Since the paper states that "the only change is this single step," this choice narrows the difference down to that step, and the scale difference is absorbed by the learning rate sweep.

Learning Rate Sweep and Main Run (Preset Rules)

1. Sweep: 50M tokens, batch 64 sequences (32,768 tokens, 1,525 steps), seed 0. Candidates: AdamW {1e-3, 2e-3, 4e-3, 8e-3}, Muon {0.01, 0.02, 0.04, 0.08}, MUON+ {0.02, 0.04, 0.08, 0.16}. Pick the lowest final validation loss, and if it is at the edge of the grid, run one more point in that direction (only once).
2. Main run: 100M tokens (3,051 steps), batch 64 sequences, respective optimal learning rates, seeds 0, 1, and 2.
3. Large batch: 100M tokens, batch 256 sequences (131,072 tokens, 762 steps), AdamW and Muon, learning rates at 1x and 2x of the batch 64 optimal values. One additional seed for the better performing one.
4. Decision criterion: If the final loss difference is greater than the sum of the two optimizers' seed standard deviations, it is "better"; if smaller, it is "indistinguishable".

This sequence was automatically chained by the execution script according to the rules, in order to eliminate any opportunity to intervene after seeing the results.

Environment

Executed on a single 8GB laptop GPU, PyTorch 2.9 (CUDA 12.8), training jobs on Kubernetes (Kubeflow Trainer). At the end of each run, validation loss curves (~40 points), throughput (excluding the first 11 steps), and maximum GPU memory were recorded.

3. Learning Rate Sweep: Different Optimizers Have Different "Good Neighborhoods"

Three things are visible.

• The optimal values themselves differ by about tenfold. AdamW is at 1e-3, while Muon is at 0.01. This is due to the "scale-correction convention" seen in Part 1. Comparing them with the same learning rate magnitude guarantees that one side will suffer.
• AdamW degraded by 0.10 when straying just twice away from optimal, and by 0.59 at four times. Muon showed differences of 0.005 and 0.032 at half (0.005) and twice (0.02) the optimal value, respectively. At this scale, Muon was less sensitive to the learning rate. Looking at AdamW's curve, it was consistently slow from the beginning without spikes. This might be due to a short warmup, which was checked separately in Section 6.
• MUON+ followed almost the same curve as Muon. However, MUON+'s optimum (0.01) was at the edge of the expanded grid, leaving open the possibility that 0.005 might be better (Section 6).

4. Main Run: 100M Tokens, 3 Seeds

Decision (Preset criterion: difference > sum of two standard deviations)

• AdamW − Muon = 0.081, sum of standard deviations 0.006 → Muon is better. In all three seeds, Muon was lower than AdamW by 0.07 or more.
• AdamW − MUON+ = 0.068 → MUON+ is also better than AdamW.
• Muon − MUON+ = −0.013, sum of standard deviations 0.006 → At this scale and learning rate, Muon is slightly better than MUON+. This is the opposite direction of the paper's LLaMA-60M results (where MUON+ was slightly better, improving PPL from 25.75 to 25.25). The fact that the learning rate was at the edge of the grid is revisited in Section 6.

How Many Tokens Are Required for the Same Loss

Because the constant learning rate phase (2M–80M tokens) of the curves places both optimizers on the same schedule, we can compare the tokens required to reach a given loss for the first time. This is the same definition as the token ratio R = T(AdamW) / T(Muon) from 2505.02222.

AdamW required 1.4–1.6x the tokens of Muon to reach the same loss, and the ratio gradually decreased as training progressed. Including the final decay phase, Muon reached AdamW's final loss (3.490) at 90.2M tokens (across all seed sets). However, since the loss drops steeply following the schedule shape during the decay phase, this "approximate 10% savings" should be read only as a lower bound, because running Muon from the start with a 90M token budget would have already finished decaying by that point, yielding an even lower loss (that run is unmeasured).

The advantage in this experiment is much larger than the paper's "10–15% fewer tokens for the same loss" (2505.02222, hundreds of millions to billions of parameters, several times Chinchilla optimal tokens). This is likely because the model is small (34M) and training is very short at roughly 3 tokens per parameter. As seen in the table above, the ratio decreased the longer training lasted.

Time and Memory

• Throughput for Muon was 5.4% lower than AdamW (94,170 vs 99,583 tokens/sec), due to the Newton–Schulz matrix multiplication incurred at every step. MUON+'s single normalization step added another 0.6%.
• Even so, in terms of elapsed time, Muon comes out ahead. The training time required to reach AdamW's final loss was 15.9 minutes for Muon and 16.5 minutes for AdamW (which had to run to completion).
• GPU memory was 0.09GB lower for Muon. For each hidden matrix, AdamW maintains two states (first and second moments), while Muon maintains one (momentum). This matches 25.17M parameters × 4 bytes ≈ 0.1GB.

5. Large Batch: The Gap Widened

We increased the batch size 4x (131,072 tokens) and trained on the same 100M tokens in 762 steps. For the learning rate, we tested 1x and 2x the optimal value of the 64-batch and picked the better one (both favored 1x), and ran one more seed with that value.

Using the same tokens with a quarter of the steps hurts both, but the magnitude of the penalty differs. AdamW worsened by 0.24 and Muon by 0.10. The gap widened 2.7x from 0.081 to 0.219. The token ratio in the constant learning rate phase shows the same trend. The token ratio to reach a loss of 4.2 was 1.63 for batch 64 and 1.78 for batch 256 (at 4.0, it was 1.59 vs. 1.76). This aligns with "Muon's data efficiency is maintained or increases at large batch sizes" (2505.02222) and "The larger the batch, the wider the AdamW–Muon gap" (2608.25990 Table 5: 0.048 → 0.110).

As a bonus, the throughput overhead of Muon decreased to 1.5% at large batch sizes (99,604 vs. 101,120 tokens/sec). Because Newton–Schulz runs once per step, reducing the number of steps also reduces its share of overhead.

6. Post-hoc Checks: Three Questions Arising from the Results

The following five runs were decided after seeing the results above. Therefore, they do not change the above judgments and are used solely to check "how much we can trust those judgments." All use a single seed.

• When lowering its learning rate further, MUON+ slightly outperformed Muon(0.006 at 50M tokens). Section 4's statement that "Muon is better than MUON+" is likely due to an under-tuned MUON+ learning rate. However, since the difference is about twice the seed variance (standard deviation of ~0.003), the honest conclusion at this scale is that "the two are roughly equal."
• When increasing the warmup for AdamW, the optimal learning rate shifted to 2e-3, but the achieved loss was nearly identical(3.7235 vs. 3.7270). A short warmup did not impair AdamW's best performance.
• Increasing the warmup at large batch sizes reduced the gap from 0.219 to 0.189. The short warmup put AdamW at a disadvantage of about 0.02, but the conclusion that the gap is more than twice as large as the batch 64 gap (0.081) remains unchanged.

7. Interpretation

• Even at this scale, Muon clearly won. Even after tuning the learning rate separately, matching the actual weight decay intensity, and keeping the auxiliary AdamW identical, the difference exceeded ten times the seed variance.
• The advantage was greater in the "short and large batch" regime. The earlier in training and the larger the batch, the greater the difference. This aligns with the finding in Part 1 that spending a step evenly across all directions is more valuable when the step count is low. Conversely, this suggests that the difference will diminish in longer training runs, which is consistent with the 10–15% figures mentioned in the paper.
• The gains from MUON+ were indistinguishable from noise at this scale. At the learning rate chosen by the predefined sweep, it was actually 0.013 worse, and when the learning rate was lowered in the post-hoc check, it led by 0.006. In the paper as well, the gain for the 60M model was small (0.5 PPL). A "normalization step" is a modification that requires retuning the learning rate to be effective, and that effect is small in smaller models. The difference from the paper's setup is that Nesterov and magnitude correction were kept identical to Muon in an effort to narrow the difference down to just the normalization step.

8. Limitations

• One model (34M), one dataset (FineWeb-Edu English), 100M tokens (approx. 3x parameters). This is not an experiment to confirm or refute the conclusions of papers involving hundreds of millions to hundreds of billions of parameters and long training schedules.
• The learning rate was chosen at 50M tokens and used up to 100M tokens. For large batches, only two learning rate points were checked.
• Warmup (2%), schedule, beta, and weight decay were not tuned separately for each optimizer. Only the learning rate was tuned.
• Large batches and MUON+ had two or three seeds, while the sweeps used a single seed.
• "Tokens to a given loss" was measured only during the constant learning rate phase. Precise savings ratios including the cooldown phase require additional runs with varied budgets (not measured).

Raw Data

• Code: train_gpt.py(model, optimizer, training loop), driver.py(in order of predefined sweep, main run, large batch), prep_data.py(data, tokenizer), analyze_a.py·token_ratio.py(tables).
• Raw data: per run, log.csv(tokens per evaluation point, validation loss, time) and done.json(final loss, throughput, memory), preregistration document PLAN.md.
• Data: FineWeb-Edu sample-10BT first shard row chunks 0–213 (ODC-By 1.0). Model and optimizer code were written from scratch.