Does RL Selectively Filter: GRPO and pass@k Measured on a 0.5B Model

Status: Draft (2026-10-31), measured. All training and evaluations were run directly by the author on a single 8GB laptop GPU. The conditions and decision criteria were written down before viewing the results, and the two changes made before training (task and grading) are explained along with their reasons in Section 4. Paper figures were transcribed from the original arXiv texts along with their table and figure numbers. Writing Rules: Measured data is the default; unmeasured items are marked as "Unmeasured", and unverified items are marked as "Unverified".

Table of Contents

• 1. Question: Does RL Create New Capabilities
• 2. What the Papers Say
  • GRPO: Comparing Answers for the Same Problem Without a Value Network
  • Intersection of pass@k Curves (2504.13837)
  • RL Changes Only 1–4% of Tokens (2605.06241)
  • Even with a Single Example (2504.20571)
  • It May Differ for Smaller Models (2607.16097)
• 3. Experimental Design
• 4. Two Changes Made Before Training
• 5. Results
  • Learning Curves
  • pass@k: Using Two Scoring Methods
  • Pre-determined Criteria (Problem-level Bootstrap, 95% Interval)
  • How the Same Problems Are Solved
• 6. Interpretation
• 7. Limitations
• Raw Data and References

1. Question: Does RL Create New Capabilities

Since DeepSeek-R1 in early 2025, reinforcement learning has become the standard approach for building reasoning models on "problems with mechanically verifiable correct answers" (such as mathematics and code). This is referred to as RLVR (RL with verifiable rewards). The model generates multiple answers, increases the probability of correct answers, and decreases the probability of incorrect ones.

However, whether this process creates capabilities that did not exist before, or simply makes correct answers that the model could already occasionally produce appear more frequently, is a different story. The tool used to measure this distinction is pass@k.

• pass@1: The probability of getting it right in a single sample. This is the commonly used accuracy metric.
• pass@k: The probability of getting at least one right when sampling k times. If an answer checker is available, this represents the upper bound attainable via best-of-k (sampling multiple times and picking the correct one).

If RL merely "filters and selects," pass@1 will increase, but pass@k (for large k) will remain nearly unchanged, because the base model will eventually produce that answer if sampled enough times. If it creates new capabilities, pass@k must also increase.

2. What the Papers Say

GRPO: Comparing Answers for the Same Problem Without a Value Network

Proposed in DeepSeekMath (arXiv 2402.03300), GRPO is a method that removes the value network (critic) from PPO. It samples G answers for a single problem, subtracts the group meanfrom each answer's reward, divides by the standard deviation, and uses the resulting value as the advantage for all tokens in that answer. The different answers to the same problem serve as the baseline. The original paper's settings were a learning rate of 1e-6, a KL coefficient of 0.04, and 64 samples per problem. Figure 7 in the same paper already notes the observation that "RL increases Maj@K but does not increase Pass@K."

Dr. GRPO (arXiv 2503.20783) pointed out that the two divisions in the GRPO formula introduce bias. Dividing by answer length penalizes longer incorrect answers less, causing incorrect answers to become longer, while dividing by the group standard deviation places excessive weight on problems that are either too easy or too hard. Dr. GRPO proposes removing both and dividing by a constant instead. This experiment follows Dr. GRPO.

Intersection of pass@k Curves (2504.13837)

Yue et al., in "Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?", increased k to hundreds or thousands across various models and tasks and plotted pass@k. When k is small, the RL model wins, but when k is large, the base model catches up and surpasses it. The problems solved by the RL model were almost a subset of those solved by the base model. The measurement setup used a temperature of 0.6, top-p of 0.95, and n samples per problem (where n = the largest k in the curve) calculated using the unbiased estimator 1 − C(n−c, k)/C(n, k) , which this article also adopts. However, looking at Table 4 in the same paper, the MATH500 pass@256 for Qwen2.5-7B was 96.2 for the base model, 97.2 at 150 steps of GRPO, 96.0 at 300 steps, and 95.4 at 450 steps. At early training checkpoints, the upper bound actually increased slightly, and decreased as training continued longer. This implies that in short training runs like our current experiment, the upper bound can indeed increase.

RL Changes Only 1–4% of Tokens (2605.06241)

The 2026 paper "Rethinking RL for LLM Reasoning: It's Sparse Policy Selection, Not Capability Learning" examined the same question at the token level. Comparing greedy decoding before and after RL reveals that the locations where RL selects different tokens account for only 1–4% of the total, and the tokens selected by the RL model at those positions were almost always within the top 5 candidate tokens of the base model (Table 2, mean rank around 2). The modified positions were concentrated at decision points where the base model hesitates (high entropy). Based on this observation, the authors built ReasonMaxxer, which corrects choices at decision points using just 50 problemsand offline rollouts without RL, and report that it achieved performance comparable to full RL (Table 3). Two points of caution when reading the original text: ReasonMaxxer uses offline training weighted by advantages rather than GRPO, and the statement in Section 3.1 that "not a single shifted position exists" does not align with the 0.01–0.12% figure in Table 2. In Table 3 of the same paper, applying GRPO to Qwen3-0.6B resulted in a modest increase on MATH500 from 0.338 to 0.372.

Even with a single example (2504.20571)

1-shot RLVR applies to Qwen2.5-Math-1.5B by a single problemand reports that after duplicating it to perform RLVR, MATH500 rose from 36.0 to 73.6, which was equal to the result of training on the 1.2k problems containing that problem. The selection criterion was a problem with high variance in accuracy during training, and problems that the base could never solve or had incorrect labels were ineffective. Comparing it to a baseline given only format rewards, the authors isolated the "portion for fixing the format," and the fact that this portion is large in small model experiments connects to the core lesson of this experiment.

It may be different for small models (2607.16097)

"Understanding Reasoning from Pretraining to Post-Training," which reviewed everything from pretraining to RL for a 5M–1B model using chess, viewed RL not as evenly sharpening, but as a redistribution combining answer amplification, rare answer discovery, and wrong answer amplification. Also, in the 20M model, pass@16 rose quickly in the early stages of RL (Figure 2). This is a warning that when reproducing with a small model, one should not expect a crossover to necessarily occur.

3. Experimental Design

We scaled down the size so it can be run on a single 8GB laptop GPU.

• Model: Qwen2.5-0.5B-Instruct (Apache-2.0). The bf16 weights are frozen, and only LoRA (rank 32, alpha=64, all linear layers in attention and MLP) is trained. Full fine-tuning requires about 8GB including AdamW states, so it does not fit. LoRA and GRPO were written from scratch without libraries (totaling just under 300 lines). One of the reasons is to show the loss line by line on the blog.
• Task: Three-digit × two-digit − three-digit arithmetic (e.g., 413 * 41 - 882). The problems were generated with code (4,000 for training, 128 for testing, no overlap). The prompt requires "<think> solution inside, <answer> answer inside".
• Reward: <answer> : 1 if the last integer inside is the correct answer, 0 otherwise. Format scores are not given separately.
• GRPO(Dr. GRPO style): 16 problems × 8 answers per step (temperature 1.0, maximum 384 tokens). Advantage = reward − group mean (not divided by standard deviation). Loss = −Σ advantage × token log-probability ÷ (16 × 8 × 384). No KL term. Since it is sampled once and updated once, PPO's ratio and clipping do not operate. AdamW (β 0.9·0.99), 150 steps.

The core of the loss is a few lines.

rs = np.array([score(t, p) for t, _ in outs])     # 그룹 8개의 보상(0 또는 1)
adv = rs - rs.mean()                              # Dr. GRPO: 표준편차로 나누지 않는다
...
logp, mask = completion_logprobs(model, tok, prompts, completions)   # 응답 토큰의 log π
loss = -(A * logp * mask).sum() / (16 * 8 * 384)  # 응답 길이로 나누지 않는다(상수)
loss.backward()

Three Conditions and Baseline

For the 16-problem and 1-problem conditions, the first 128 training problems were solved 16 times each by the base (temperature 1.0) to select problems with mixed pass rates. Among the 128, 31 were between 0 and 1. This is the same idea as the "edge of competence" in 2605.06241 and the "problems that occasionally yield correct answers" in 1-shot RLVR.

Learning Rate: Since there is no basis in the papers for the learning rate when performing GRPO with LoRA, we briefly looked at {1e-5, 5e-5, 2e-4} with 30 steps of the "Many problems" condition and used the value with the highest average reward in the last 10 steps identically for all three conditions. The results were 1e-5: 0.38, 5e-5: 0.60, and 2e-4: 0.00. At 2e-4, the policy collapsed and all answers repeated the same words until the end of 384 tokens.

Evaluation: 32 samples per 128 test problems (temperature 0.6, top-p 0.95), pass@1–32 using the U-estimator. Base and 75·150 steps for each condition. Judgment produced a 95% interval for the (RL − base) difference via problem-level bootstrapping (2,000 times). Pre-determined criteria are "pass@1 rises = interval lower bound > 0" and "upper bound remains unchanged = pass@32 difference interval includes 0 or the difference is less than 1/3 of the pass@1 difference".

4. Two Things Changed Before Training

Before seeing the results, I changed the predetermined conditions twice before training. Both changes were made before running a single step of RL, and the times and reasons for the changes were documented in the pre-registration record.

1. Task: Countdown → Arithmetic. The initial plan was countdown (combining three numbers with arithmetic operations to reach a target number). Measuring the baseline pass rate, the 0.5B model almost never got a single problem right out of 32 problems × 16 samples (pass@16 0–3%). If all 8 groups are 0, the advantage is 0, so GRPO has no signal to learn from (p^G + (1−p)^G analyzed in 2609.25463). Therefore, I changed it to arithmetic.

2. Splitting grading into two, and moving to harder arithmetic. For two-digit × two-digit − three-digit arithmetic, the baseline pass@1 was 0.10, but when graded by ignoring the answer format and looking only at the "last integer in the response," it was 0.62, and pass@32 was 1.0. The model already knew how to calculate; what it lacked was <answer> the tag format (format compliance rate 25%). This task cannot answer the question "Does RL raise the upper limit of capability?" (it is already 1.0). So I changed two things.

• I added lenient gradingto the evaluation (where the answer inside the tags is correct or the last integer in the response is correct) to separate "format" and "calculation." The training reward remains strict grading. Both judgments are produced, but the decision on whether "RL selects" places more weight on lenient grading, because the format is an output protocol rather than a capability. This sequence was also recorded before seeing the results.
• I made the task harder: three-digit × two-digit − three-digit. The baseline lenient pass@1 is 0.18 and pass@32 is 0.81, leaving room for calculation capability to improve.

Additionally, I found that a repetition penalty of 1.1 was present in the model file's generation_config.json and interfering with sampling, so I fixed it to 1.0 before training. Since the RL gradient formulation assumes that samples come directly from the policy, hidden sampling settings quietly alter the results. There were also instances where pods died twice due to memory limits, but this was unrelated to the conditions.

5. Results

Learning curves

The 16-problem and 1-problem conditions solved all training problems within 50 steps. After that, all 8 groups were 1, so the advantage was 0, meaning there was no further signal to learn from.

pass@k: With two types of grading

Models after RL all adhered to the format nearly 100%, making strict grading and lenient grading identical. Only the baseline shows a difference.

Pre-determined judgment (problem-level bootstrap, 95% interval)

Under lenient grading, the number of problems solved correctly at least once out of 32 (coverage) decreased from 107 in the baseline to 103 for many problems at 75 steps (12 solved only by RL, 16 solved only by baseline), and to 87 (12 / 32) at 150 steps. It was 25 for 16 problems and 38 for 1 problem.

How the same problem is solved

Response to the first test problem 413 * 41 - 882(correct answer 16051) (150 steps, temperature 0.6, first of 32, line breaks indicated by /).

• Baseline(0 format-compliant correct answers out of 32, 7 leniently correct): "1) Calculate 413 × 41 / 413 × 41 = 17073 / 2) Subtract 882..." It breaks down steps but sometimes messes up the multiplication itself in one go, and fails to put the answer in tags.
• Many problems(27 correct out of 32): "413 * 41 = 413 * (40 + 1) = 413 * 40 + 413 * 1 = 16520 + 413 = 16933 / 16933 − 882 = 16051 / ..." It chose the approach of breaking down the multiplication. This solving method is one the baseline occasionally produced as well.
• 16 problems(0 correct): "413 * 41 = 16613 / 16613 − 882 = <answer> 15731 </answer> </think> </think> ..." It performs the multiplication in a single line and repeats the closing tag after the answer up to the maximum length.
• 1 problem(0 correct): "413 * 41 = 16633 / 16633 − 882 = <answer> 15751 </answer>" Short and clean, but fails because it does the multiplication all at once.

6. Interpretation

1. GRPO with many problems "selected." pass@1 viewed as calculation capability increased significantly from 0.25 to 0.66, but the probability of getting at least one right when sampled 32 times (pass@32) was statistically tied at 0.84 to 0.81. As k is increased, the baseline catches up; the 75-step model was overtaken by the baseline between k=16 and 32, and the 150-step model between k=8 and 16 (Figure). The crossover observed by Yue et al. appeared in the 0.5B model and arithmetic task as well. Extending training to 150 steps caused pass@32 to drop significantly to 0.68, and coverage also shrank from 107 to 87 problems. This is in the same direction as "longer training decreases pass@256" in Table 4 of the same paper. What was selected is also visible in the responses: breaking down multiplication into (40 + 1) is a solution already present within the baseline distribution. This aligns with the figure that "choices at decision points among top baseline candidates are altered," as stated by Akgül et al. (2605.06241) (token-level analysis was not measured).

2. Looking only at strict grading would have led to a different conclusion. Under strict grading, pass@1 increased across all three conditions, and the many-problems condition saw pass@32 rise sharply from 0.26 to 0.81. These are numbers that could be interpreted as "RL raised the upper limit." However, most of that was <answer> the share gained from learning to use tags. The baseline's format compliance rate was 16%, whereas it was 100% after RL. This is why 1-shot RLVR uses a separate "format reward baseline" and why Dr. GRPO pointed out template issues. The smaller the model, the larger the share format occupies.

3. GRPO with few problems failed in this setup. The 16-problem and 1-problem conditions learned the format (strict pass@1 +0.12, +0.16) but saw their calculation capability actually drop (lenient pass@1 −0.11, −0.07), and diversity collapsed, pulling pass@32 down to 0.20 and 0.30. After correctly solving all training problems within 50 steps, the rewards were all 1, leaving no signal. In the meantime, they got trapped in bad habits like short solutions doing "single-line multiplication" or "repeating tags after the answer." This does not refute the findings of Akgül et al. (2605.06241) that "dozens of problems are enough" or those of 1-shot RLVR. The conditions are different.

• 1-shot RLVR used KL (0.001) and an entropy bonus, reporting that the entropy term boosts generalization after saturation. This experiment uses neither.
• ReasonMaxxer (2605.06241) uses weighted training on offline rollouts rather than GRPO, anchoring tokens outside of decision points to the baseline distribution (KL anchor). The success of "few problems" relied jointly on this anchoring mechanism.
• The learning rate (5e-5) was chosen for the many-problems condition. It may be too large for conditions seeing the same problems 150 times.
• The model is a 0.5B general instruction model and the task is arithmetic. Many of the papers feature math-specialized models of 1.5B or larger.

Ultimately, what this small experiment shows is that "doing RL with few problems seems to require an anchoring mechanism to keep the policy from drifting too far from the base." This is a hypothesis for the next experiment and is not yet measured.

7. Limitations

• There is a single seed (one per condition). The bootstrap intervals capture only problem sampling variance and not training variance.
• One model (0.5B instruct model), one task (synthetic arithmetic), LoRA (rank 32), 150 steps. This differs from the scale of the papers (1.5B–671B, thousands of steps).
• Lenient grading uses the "last integer in the response," which may slightly underestimate the baseline's calculation capability (writing another number after the answer is counted as incorrect).
• Control training with KL, entropy, or format reward baselines was not performed (unmeasured). Learning rates were not tailored per condition either.
• pass@k goes up to k=32. While crossovers appeared at k=8–16, which was sufficient for the conclusion, the shape at larger k remains unmeasured.

Raw Data and References

• Code: grpo.py(LoRA, GRPO, evaluation, problem selection), countdown.py(problem generation and grading), pipeline.py(pre-determined sequence), analyze_b.py(judgment).
• Raw data: per-evaluation eval_*.json(correct counts per problem, format compliance counts, response samples), training train_log.csv, learning rate selection lr_choice.json, pre-registration PLAN.md(changes 1–4).
• Model: Qwen2.5-0.5B-Instruct (Apache-2.0, revision 7ae5576). Problems are generated programmatically, so no downloaded dataset exists.
• Shao et al., "DeepSeekMath", arXiv 2402.03300. DeepSeek-AI, "DeepSeek-R1", arXiv 2501.12948.
• Liu et al., "Understanding R1-Zero-Like Training: A Critical Perspective" (Dr. GRPO), arXiv 2503.20783.
• Yue et al., "Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?", arXiv 2504.13837.
• Wang et al., "Reinforcement Learning for Reasoning in LLMs with One Training Example", arXiv 2504.20571.
• Akgül et al., "Rethinking RL for LLM Reasoning: It's Sparse Policy Selection, Not Capability Learning", arXiv 2605.06241.
• Shen et al., "Understanding Reasoning from Pretraining to Post-Training", arXiv 2607.16097.
• Gholipour et al., "Rollout Efficiency in RL for Reasoning LLMs", arXiv 2609.25463.