Classical Methods and Reinforcement Learning: What to Solve with Equations and What to Leave to Learning

Status: Draft (2026-10-18), empirically measured. Numbers are simulation values calculated using Python 3.9.6 and numpy 2.0.2 on a laptop (Apple M1 Max, macOS 26.5). Raw data and code: Working directory measure/robot-arm-basics/'s m10_learn.py, out/m10_learn.json. The "real world" is also a simulation, and there are no actual robot experiments (hardware not empirically measured). Paper numbers were verified from each original text (arXiv versions indicate the version number). Reference materials were accessed on 2026-10-18. Writing rules: Empirically measured data is the default; unmeasured items are marked as "unmeasured", and unverified items are marked as "unverified".

Table of Contents

• 1. Intuition: Calculate What You Know, Learn What You Don't Know
• 2. Figure: Boundaries of Four Systems
• 3. Comparison of Papers
• 4. Empirical Measurement: The Same Question as a Smaller Problem
• 5. Where to Draw the Boundary: Proposals of This Series
• 6. Limitations
• 7. Concluding Part 1
• References

1. Intuition: Calculate What You Know, Learn What You Don't Know

Parts 1 through 9 were all about equations. Ball flight was Newton's laws, racket posture was geometry, trajectories were polynomials, and control was inverse dynamics. Equations work from the very first ball without data, and can explain why they failed. Instead, they fail precisely as much as the model is wrong. In Part 7, a 0.05 error in the racket restitution coefficient caused the landing point to miss by 13 cm.

Reinforcement learning (RL) is the opposite. Since it learns through trial and error without a model, it is not bound by model errors, but it requires many hits and it is difficult to inspect what was learned.

All studies building table tennis robots mix the two. The question is not "whether to mix or not" but **"where to draw the boundary."**

2. Figure: Boundaries of Four Systems

3. Comparison of Papers

This is the configuration verified by opening each original text. Direct comparison is not possible because the conditions next to the numbers differ.

In summary, as the boundary moves to the right, the share of learning increases.

• Mülling et al.kept physics (EKF prediction) and motion shapes (demonstration) as they were, and used reinforcement learning only to determine "which motions to mix and in what proportions." They improved the success rate from 69% to 97% over 150 hits.
• Koç et al.used almost no learning, estimating ball model constants from human demonstration data and then solving trajectories via optimization (optimization average 25 ms). This is an example of overcoming the limitations of the fixed virtual hitting plane mentioned in Part 1 through optimization.
• Büchler et al.learned from scratch without prior knowledge on an arm where models are difficult to build, such as pneumatic muscles. Instead of flying balls physically, they replayed 100 recorded trajectories in simulation (HYSR), while the arm moved physically. They performed about 15,000 hits for the return task, taking about 14 hours.
• D'Ambrosio et al.learned each skill individually in MuJoCo simulation (using BGS, an evolution strategies variant without gradients, which they noted yielded smoother motions than PPO or SAC), and do not run a flight model in real time. Instead, they combined a spin classifier, per-skill performance logs, hand-crafted rules, and opponent preference learning in the high-level controller that selects skills. The chosen skill is determined within 20 ms.

There is also a workflow of correcting models through learning. Sony AI 2026 re-fitted the ball's drag, Magnus, and bounce models using approximately 58,000 trajectories gathered by 9 cameras, and added neural network residuals on top of the physical formulas for the racket model. They report that the landing point center error decreased from 0.37 m of the baseline model to 0.15 m.

4. Empirical Measurement: The Same Question as a Smaller Problem

Because the papers differ in robots, balls, and evaluations, it is impossible to directly compare "how many hits are needed if the boundary is drawn at a certain place." Therefore, we took the hitting problem from Part 7 and conducted a small experiment.

Setup. The racket in the "real world" differs from the model. Restitution coefficient e = 0.72 (model 0.8), tangential velocity preservation rate k = 0.88 (model 1.0, tangential velocity reduced by 12% due to rubber friction). The balls are 300 actual shots chosen from the dataset in Part 1 (200 for training, 100 for testing), and the goal is to land the ball on the opponent's side at (0.3, 0.9) m in 0.6 seconds.

Three Methods.

• (a) Equations only: Uses the closed-form solution from Part 7 with the nominal model (e = 0.8, k = 1).
• (b) Learning physical constants only: Observe incoming and outgoing ball velocities while hitting N times (camera noise σ 0.05 m/s), and re-fit the two numbers e and k using least squares. The racket is continuously calculated via closed-form solutions. Since the normal direction formula is linear with respect to e (v_out·n − v_r·n = e·(v_r·n − v_in·n)), it can be solved directly.
• (c) Policy search without a model: A linear policy (18 parameters) receiving ball state (3 velocities, 2 positions, 1 constant) to output two angles of the racket normal and the racket speed is found using the cross-entropy method (CEM). Each generation evaluates 40 candidates, hitting 10 balls per candidate, and ranks them by average landing error.

# (b) 한 번 칠 때마다 e, k 를 최소제곱으로 다시 맞춘다
n, vrn = racket_general(V, v_out_wanted, e, k)        # 지금 추정값으로 닫힌 해
vout = true_racket(V, n, vrn)                          # 실제 세계(모델과 다름)
Vm, Om = V + noise(), vout + noise()                   # 관측
A_e.append(vrn - Vm @ n);  b_e.append(Om @ n - vrn)    # e 에 대해 선형인 식
e = dot(A_e, b_e) / dot(A_e, A_e)

(b) is the median of 10 seeds, and the e and k estimated after 100 hits were 0.720 and 0.876 (true values 0.72, 0.88).

• When boundaries are drawn well, learning becomes surprisingly cheap. Because equations inform us that there are only two unknown numbers, hitting just a single ball reduced the error from 24.5 cm to 2.1 cm.
• Discarding the model is expensive. Solving the same problem without a model missed by over 1 m even after 2,000 hits, and plateaued around 25 cm after 8,000 hits. This is because a linear policy cannot capture the non-linear relationship of "ball velocity → racket angle." Scaling up the policy to a neural network might improve things, but it would require correspondingly more hits (unmeasured).
• In a real robot, 20,000 ball hits take more than a day just for a human to pick up the balls. This cost is why Büchler et al. replayed balls in simulation and D'Ambrosio et al. learned skills in simulation. When learning in simulation, however, any mismatch in the simulator's constants causes deviations from reality, ultimately bringing back the task of "fitting models with data" (Sony AI 2026).

This experiment is intentionally simple. The real world is also a simulation, and the discrepancy from the model is merely "constants within the model," which favors (b). In cases where the form of the model itself is wrong (e.g., the model completely lacks the effect of rubber imparting spin), residual errors remain no matter how much constants are adjusted, and learning those residuals is what Sony AI 2026's neural network residuals do.

5. Where to Draw the Boundary: Proposals of This Series

As a starting point for the first open-source hardware robot, we propose the following. This may change through experiments.

There is only one principle: If equations can narrow down "what is unknown," learn only that narrowed-down part. Actuators whose models are difficult to build (pneumatic muscles) or decisions without ground truth (strategy) are better left to learning from the start.

6. Limitations

• CEM is among the simplest policy search methods in reinforcement learning. We did not compare it against gradient-based methods like PPO, neural network policies, or pre-training in simulation followed by fine-tuning on real hardware (unmeasured).
• Real-world differences were reduced to just two constants. Comparisons in a world with spin, individual ball variations, and observation delays are unmeasured.
• Numbers for Büchler et al. are based on arXiv v1. We were unable to check the final T-RO 2022 version (unconfirmed). Since we could not access the original biomimetic approach paper by Mülling et al. 2011, the four-stage motion classification was verified only through the figure descriptions in their 2013 paper.

7. Conclusion of Part 1

Over ten articles, we have calculated through the seven steps required to hit a single table tennis ball. Here is a collection of key numbers to remember when transitioning to open hardware.

The next step is to remeasure these numbers with actual hardware and in the simulator. Before that, articles 11 to 15 will continue by implementing the principles of reinforcement learning (MDP, PPO, reward design and action space, sim-to-real transfer, evaluation) through toy problems.

Titi's Word. If you try to learn everything, hitting a ball 20,000 times won't be enough. Leave what you know to the equations and only learn the two unknown numbers, so ten times is plenty!

References

All materials were accessed on 2026-10-18.

• Mülling, Kober, Kroemer, Peters, Learning to select and generalize striking movements in robot table tennis, IJRR 32(3) (2013) 263–279, Author's preprint, Sections 3.1–3.3, Figure 7.
• Koç, Maeda, Peters, Online optimal trajectory generation for robot table tennis, RAS 105 (2018) 121–137, Author's preprint, Section 6, Figure 11, Table A.3.
• Büchler, Guist, Calandra, Berenz, Schölkopf, Peters, Learning to Play Table Tennis From Scratch using Muscular Robots, arXiv:2006.05935 v1 (T-RO 2022 final version unconfirmed).
• D'Ambrosio et al., Achieving Human Level Competitive Robot Table Tennis, arXiv:2408.03906, Sections II–III, Appendix Table XIV.
• D'Ambrosio et al., Robotic Table Tennis: A Case Study into a High Speed Learning System, arXiv:2309.03315.
• Conti et al. (Sony AI), Physics Models for Sim-to-Real Transfer in Professional-Level Robot Table Tennis, arXiv:2606.28805.
• Google DeepMind, competitive_robot_table_tennis, CC-BY 4.0.