Reinforcement Learning Evaluation: Training Balls and Evaluation Balls, Medians Across Multiple Seeds, Final Value Reporting, and Baselines

Status: Draft (2026-10-18), empirically measured. Numbers were calculated by re-reading training results from posts 11 to 14 (measure/robot-arm-basics/out/rl/*_s{0,1,2}.json, 16 configurations × 3 seeds). The environment is a laptop (Apple M1 Max, macOS 26.5), Python 3.9.6, and numpy 2.0.2. There are no actual robot experiments (hardware not measured). References were accessed on 2026-10-18. Writing rules: Measured data is the default; unmeasured items are marked as "Unmeasured," and unverified items are marked as "Unverified."

Table of Contents

• 1. Intuition: A Single Good Curve Can Lie
• 2. Rule 1: Separate Training Balls and Evaluation Balls
• 3. Rule 2: Run Multiple Seeds and Report Medians and Ranges
• 4. Rule 3: Report the Final Value, Not the Best Moment
• 5. Rule 4: Place Alongside Baselines
• 6. Rule 5: Aim for the Real Goal, Reproducibly
• 7. Next: A Real Table Tennis Arm
• 8. Limitations
• 9. Conclusion
• References

1. Intuition: A Single Good Curve Can Lie

In reinforcement learning, even with the same code and configuration, results vary widely depending on the random seed. Because learning curves fluctuate up and down, picking the 「highest point」 inflates the perceived skill level. Also, there is no guarantee that a policy that performs well in the simulator will perform well in the physical world (Post 14).

Henderson et al. (2017) pointed out that the non-determinism of benchmark environments and the inherent variance of methods make it difficult to reproduce deep reinforcement learning results. Agarwal et al. (2021) suggested using robust metrics like interval estimates and interquartile mean (IQM), along with performance distribution plots, when producing results from a small number of runs (abstracts for both papers). This post reviews the five rules maintained by this series using the actual numbers from posts 11 to 14.

2. Rule 1: Separate Training Balls and Evaluation Balls

Out of 7,315 balls passing near the racket arc, 1,000 were set aside from the beginning and not used for training. The trained policies were evaluated separately on 1,000 training balls and 1,000 evaluation balls.

The reason the difference was small even when dividing the balls is that the two ball bundles came from the same distribution. Since 6,315 balls were drawn and hit 19,000 times, it is difficult for the policy to memorize individual balls. The real danger is when the distribution changes. The same policy lost 10%p in the "real world" with slightly different physical constants. Therefore, evaluation must be done on two things: balls from the same distribution not used in training, and intentionally different distributions(different restitution, delay, faster balls, stronger spin).

3. Rule 2: Run Multiple Seeds and Report the Median and Range

If you had only run Seed 0, you might have written that "GAE λ = 1 (84.2%) is better than PPO Default (82.9%)." Looking at all three seeds, λ = 1 failed twice. Conversely, the entropy bonus has all three seeds clustered within 0.3%p, giving grounds to call it "stable." Three seeds are still few, so in parts 12–14, we did not rank settings whose ranges overlap.

4. Rule 3: Report the Final Value, Not the Best Moment

During training, evaluations were performed every 20 iterations (10 iterations for the hybrid). We compared what happens if you pick the highest value among them.

There is no difference in settings where learning steadily improves, but in fluctuating settings, the "best-performing value" inflated the result by 14 percentage points. A more fundamental problem is that this "best-performing value" was chosen using the evaluation balls. If evaluation balls are used to select checkpoints, they are no longer unseen balls. If you need to select checkpoints, pick them from a validation set separated from the training balls, and use the evaluation balls only once at the end.

5. Rule 4: Place alongside baselines

The number "83% with PPO" alone does not tell you if it is good. By presenting it alongside a random policy at 1.5%, a classical planner at 40%, and the upper bound of the same action space at 85%, you can say that it is "twice as good as a planner using a flawed model, and near the upper bound achievable with a 1-DOF racket." The fact that Paper 13's hybrid approach (46%) is equivalent to the best constant action (46%) would have been misread as "learning occurred" if there were no baseline.

6. Rule 5: Evaluate on the true objective and ensure reproducibility

• Evaluate on the true objective. The policy that won the "reward-based prize" in Paper 13 had a 98% success rate based on reward criteria, but only landed 0–9% of the balls on the table. Regardless of the training reward, evaluation was based solely on a single question: "Did it land the ball on the opponent's table?"
• Perform evaluation without stochasticity. During training, the policy samples actions from a normal distribution, but evaluation uses only the mean action. Evaluating the same policy with the same ball yields the same number.
• Verify reproducibility. The "no randomization" experiment in Paper 14 shared the same setup as the PPO baseline in Paper 12, with just one additional evaluation. The nominal simulation success rates of the two experiments matched down to the decimal point across seeds (82.9%, 76.8%, 90.9%). When seeds and code are fixed, learning in this toy environment is deterministic. Because this is not the case with real robots or GPU parallel physics (unmeasured), more seeds and interval estimation are required.
• Document the environment. We recorded the Python and numpy versions, CPU, concurrency count, training volume (episodes and environment steps), and training time along with the results (status banner of Paper 12).

7. Next: The Real Table Tennis Arm

Toy problems 11 through 15 were meant to demonstrate principles with numbers. In the same project, we are performing reinforcement learning on the MuJoCo table tennis simulator using an arm close to the real physical structure. As the results are not yet finalized at the time of writing this post, we did not include the numbers. Once the results are available, we will report them in this series as a "Real Training Results" post, following the five rules of this post (divided evaluation balls, multiple seeds, final value, baseline, true objective).

8. Limitations

• We used 3 seeds. We did not calculate interval estimates (bootstrap confidence intervals) and IQM as recommended by Agarwal et al. With 3 seeds, IQM is equivalent to the median.
• The success rate for 1,000 evaluation balls has a sampling error of about ±2.2 to 3.1%p (95%, binomial approximation, our calculation) at success rates of 50 to 85%. Differences smaller than this are not meaningful.
• "Out-of-distribution" evaluation was done only for the "Reality" case in Part 14. Distribution shift evaluations, such as faster balls or stronger spin, were not measured.

9. Conclusion

• Separate training and evaluation balls, and evaluate different distributions separately. The difference within the same distribution was around 1%p, while it was 10%p when the distribution changed.
• Use multiple seeds, and report using the median and range. Conclusions were flipped with just a single seed.
• Report the final value, not the best-performing moment. You must not select checkpoints using the evaluation ball.
• Numbers only gain meaning when placed alongside a baseline.

A word from TT. Showing just one video of a well-hit practice ball makes anyone look like a pro. Showing your real skills with balls you see for the first time, over multiple tries, right until the end is the real deal!

References

All materials were accessed on 2026-10-18.

• Henderson, Islam, Bachman, Pineau, Precup, Meger, Deep Reinforcement Learning that Matters, arXiv:1709.06560, Abstract.
• Agarwal, Schwarzer, Castro, Courville, Bellemare, Deep Reinforcement Learning at the Edge of the Statistical Precipice, arXiv:2108.13264(NeurIPS 2021), Abstract.