Trends in LLM Training Research for 2026: Optimizers, Data, Distillation, and RL

Status: Draft (2026-10-31). Paper metrics were copied directly by opening the original arXiv articles (the main text uses HTML of the same version), along with table and figure numbers. The experimental lessons in Section 7 come from the actual measurements in Parts 2 and 3 of this series. The basis for selecting papers (Hugging Face upvotes, public dataset download counts) values checked on 2026-10-31. Writing rules: Measured data is the baseline; unmeasured items are marked as "Unmeasured," and unverified items as "Unverified."

Table of Contents

• 1. Map of This Article
• 2. Data: What Is the Value of AI Text on the Web?
• 3. Distillation: On-Policy Distillation Between Same-Family Models
• 4. Small Model Recipes: Fewer Tokens, Less Money
  • IMU-1 (2602.02522)
  • Puro-2B (2608.27370)
• 5. Insights Across Categories
• 6. Checklist for Small-Scale Paper Reproductions
• 7. Lessons Learned from the Two Experiments
• References

1. Map of This Article

The previous three parts deeply explored two questions: "Is Muon better than AdamW?" (Part 1, Part 2) and "Does RL create new capabilities?" (Part 3). This article takes a step back to select high-impact training papers published in the second half of 2026 (July–October) and summarize their trends. On the pretraining side, we look at dataand small model recipes, and on the post-training side, we look at distillationinstead of RL.

Our selection method was as follows: We searched arXiv by keywords from 2026-07-01 to 10-31, sorted 122 days of Hugging Face daily papers by upvotes, and then selected papers that stood out in upvotes, public dataset usage, and code repository stars within the fields of data curation/mixing, distillation, and small model pretraining. Citation counts were not used because having been out for only 1 to 3 months, they lacked discriminatory power.

2. Data: What Is the Value of AI Text on the Web?

"How Much Is an AI Token Worth? Scaling Laws for Wild AI-Generated Web Text" (arXiv 2609.40295, University of Maryland & Pangram Labs) quantifies the most realistic problem in pretraining data today. Web crawls are already heavily mixed with AI-written text. Among tokens passing the FineWeb filter, the proportion identified as AI increased from 10.1% in June 2024 to 16.1% in June 2025 and 31.1% in August 2026 (Figure 1). Even more troubling is the fact that quality filters retain AI text better. Auditing 10,000 raw crawled documents revealed that AI documents passed the filters at rates 2.3 times higher than human documents in the FineWeb pipeline and 9.8 times higher in the DCLM pipeline (Figure 2).

The authors trained 800 models ranging from 19.9M to 973M parameters, holding human tokens constant while adding AI tokens to measure human text validation loss. The conclusion is that the sign flips. For data-scarce models with fewer than 10 human tokens per parameter, AI tokens lower the loss, but above 20 (Chinchilla optimal), adding AI tokens increases the loss. The same amount of new human tokens continues to lower the loss (Figure 3). The prediction error of the new scaling law capturing this phenomenon was about 1/5th that of the Chinchilla formulation (Table 1). There is also a lesson for evaluation: when measured with a validation set containing 22% AI text, 95.5% of harmful training appeared to be improvements (Figure 6). This implies that validation sets must be separated into human and AI text.

Points to keep in mind when reading can also be found in the original text: some of the co-authors belong to an AI text detector company, and the research was funded by that company as well. Limitations include the fact that the definition of "AI text" is determined by that company's detector, and that the law was only validated up to 973M parameters.

3. Distillation: On-Policy Distillation Between Same-Family Models

"Scaling Properties of Same-Family On-Policy Distillation" (arXiv 2609.32722) received the most Hugging Face upvotes (329) among distillation papers from July to October. On-policy distillation (OPD) rewards the student for answers it generates directly, using "teacher log probability − student log probability" per token. Unlike off-policy SFT, where the student mimics teacher-generated answers, the student learns from the path it actually takes.

Using Qwen2.5 0.5B–14B trained with math RL as teachers and distilling across staggered sizes, three things stand out. Early in training, student accuracy increases almost linearly with √KL (how far the student has drifted from initialization; linear fit R² of 0.932–0.988 across 25 runs, Figure 2). Teaching a strong student with a weak teacher results in the student outperforming the teacher (Figure 4). Furthermore, given equal accuracy, smaller teachers teach better(power-law teacher size exponent β = −0.27, Table 2). Bootstrapping with teacher answers via SFT beforehand proved significantly detrimental for the weak teacher → strong student combination (−6.6/−15.7/−19.4 points for 0.5B teacher with 3B/7B/14B students). The scope is narrow: a single model family, a single task (math), and one run per combination.

There is an interesting parallel when read alongside the experiments in Part 3: OPD between same base models yielded results within 0.5 points of directly applying RL to the student across all five sizes (Section 4.1). This aligns with the picture that if RL alters choices at rare decision points (2605.06241), teaching those choices directly via the teacher's distribution reaches a very similar place.

4. Small Model Recipes: Fewer Tokens, Less Money

IMU-1 (2602.02522)

IMU-1, written by independent researchers, trained a 430M model on 72B tokens to achieve scores close to SmolLM2-360M, which used 4T tokens (average 0.574 vs. 0.586, Table 3). It is a recipe paper that combines and validates recent techniques rather than novel methods. The architecture adds QK-Norm, per-head gating, normalized value residuals, and reduced layer-wise normalization; the optimizer uses NorMuon (re-normalizing Muon outputs at the neuron level) with Cautious Weight Decay (decaying only when update and weight signs match) and μP. Embeddings, output layers, and 1D parameters use AdamW. The learning rate schedule used WSD with the final 20% decaying in a 1−√t shape; in a 70M ablation experiment, a 20% decay yielded a loss identical to cosine (3.125, Table 7). Data was divided into three stages, raising the quality score threshold at each stage. Some inconsistencies exist within the original text, such as learning rates cited as three different values (0.0235, 0.013, 0.011), requiring verification before copying verbatim.

Puro-2B (2608.27370)

Tsinghua University's Puro-2B trained a 2B model from scratch up to 1.4T tokens using consumer GPUs, with a headline accelerator rental cost equivalent of about $6,900. It outperformed Qwen2-1.5B and approached Qwen2.5-1.5B across an average of 15 benchmarks (Tables 3 & 4). There are three core techniques.

• MuonH: Constrains weights for each matrix onto the sphere of their initial norm (re-projecting after updates) and sets weight decay to 0. This makes the effective learning rate explicit and uses 16.1% less compute for the same loss compared to tuned Muon (Section 4.3.1).
• Long Linear Decay: The larger the peak learning rate and the longer the training, the longer the decay phase needed to be (0.6B exploration, Figure 6). While seemingly conflicting with IMU-1's "20% is optimal," direct comparison is not possible due to differences in peak learning rates, token counts, and decay shapes.
• Quality Curriculum + Checkpoint Averaging: Sources were sorted in ascending order of quality score so better data came later, continued training with a constant learning rate, and then averaged the final six checkpoints.

The headline cost excludes data processing, small experiments, failed runs, and labor costs, and since the starting point for checkpoint averaging was chosen after seeing the results, its gains must be read with selection bias in mind.

5. Insights Across Categories

1. The Muon family is becoming the default choice for optimizers. The data paper (nanochat's Muon), IMU-1 (NorMuon), and Puro-2B (MuonH) all use Muon variants for hidden matrices and AdamW for embeddings, output layers, and 1D parameters. What varies is "how to rescale the orthogonalization results" (NorMuon, MUON+, MuonH).
2. The learning rate schedule is converging on an "open-ended phase + decay" pattern. WSD is standard, and the debate centers on the length and shape of the decay phase.
3. Data is handled in order of quality within sources. However, this must be read alongside the warning that those quality scores favor AI text (Section 2). Building curricula using quality scores can increase the AI proportion in later data, a point neither recipe addresses (author's deduction, unverified).
4. Post-training understanding is converging toward "making fewer changes." RL rarely changes tokens (2605.06241), similar effects can be achieved with fewer problems (2504.20571, ReasonMaxxer), and same-family distillation reaches almost the exact same places as RL (2609.32722). Therefore, rollout efficiency (2609.25463) and "what to choose as problems to learn" are shifting to the center of research.
5. Evaluation must be broken down to be seen. A mixed validation set hides the harm (Section 2), and formatting corrections look like capability improvements (Part 3).

6. Things to keep in mind when reading papers and doing small-scale reproductions

I leave the pitfalls encountered while running the two experiments as a checklist.

• The learning rate moves together with its scaling rule. Muon's "0.02" has a different meaning depending on the magnitude correction rule (Table in Part 1).
• Conditions and criteria must be written down before looking at the results. If you have to change them, leave the reason and timestamp. Even this time, the tasks for the RL experiment were changed twice before training, and without those records, it would have been impossible to distinguish them from picking after seeing the results.
• Check the default sampling settings. The model file's generation_config.json contained a repetition penalty of 1.1, and an unstated value almost sneaked into the RL samples. RL samples must match the policy as-is for the gradient formula to be correct.
• Measure format and capability separately. In Part 3, this difference changed the conclusion.
• Small models may behave differently. Like the 20M model in 2607.16097 and the 150 steps in Table 4 of Yue et al., short training and small models may not yield the same conclusions as large models.

7. What was learned from the two experiments

The numbers for both experiments are in Parts 2 and 3. Here, we tie them together with the paper trends and leave only three things.

1. Optimizer: Even at a small scale, Muon's advantages were significant, and even more so on the "short and large batch" side. When training a 34M-parameter model with 100M tokens, even after tuning the learning rate separately for each optimizer, Muon's final validation loss was 0.081 lower than AdamW's (seed set, standard deviation 0.003). To reach the same loss, AdamW required 1.4–1.6x more tokens, and when the batch size was quadrupled, the gap widened to 0.219. This is in the same direction as papers stating that "Muon preserves data efficiency in large batches," and the scale (10–15% in the papers vs. 40–60% in this experiment) is attributed to short training. On the other hand, the benefit of MUON+ was indistinguishable from noise at this scale (0.013 worse at the pre-determined learning rate, and 0.006 ahead when lowering the learning rate in a post-hoc check). The 2026 branch of "refining orthogonalization results" has a small effect, making it hard to distinguish from noise in small experiments.

2. RL: "Selection" was visible even at 0.5B, and formats can obscure that picture. GRPO performed with many problems raised calculation accuracy pass@1 from 0.25 to 0.66, but pass@32 remained unchanged and dropped with further training. Looking at the responses, it selected and reinforced the "breaking down multiplication" solutions that the base model occasionally used. This aligns with the trend of "less modification in post-training" at the end of Section 4. However, when even the answer format was graded, the same result looked like "the upper bound rose significantly as well." The smaller the model, the greater the share of format correction—a point already warned about by 1-shot RLVR and Dr. GRPO, and experiencing it firsthand proved it large enough to overturn the conclusion.

3. "Little data is enough" must be read alongside constraints. GRPO performed with 16 and 1 problems without KL and entropy terms only learned the format and worsened calculation. Papers that succeeded with few problems used KL and entropy terms (1-shot RLVR) or anchors tied to the base distribution (ReasonMaxxer) together. The final hypothesis of this series is that for "less modification" to work well, "anchoring places that should not be modified" must be present as well (unmeasured).

References

• Russell et al., "How Much Is an AI Token Worth? Scaling Laws for Wild AI-Generated Web Text", arXiv 2609.40295.
• Bao et al., "Scaling Properties of Same-Family On-Policy Distillation", arXiv 2609.32722.
• Grigorev, "IMU-1: Sample-Efficient Pre-training of Small Language Models", arXiv 2602.02522.
• Luo et al., "Puro-2B" (the rest of the title is omitted as it contains the GPU model name), arXiv 2608.27370.
• Gholipour et al., "Rollout Efficiency in RL for Reasoning LLMs", arXiv 2609.25463.
• Optimizer and RL papers are references from Parts 1–3.