The Framework of Reinforcement Learning: MDP, State, Action, Reward, Discount, Policy, and Value Function Read Through a Table Tennis Arm

Status: Draft (2026-10-18), Measured. Numbers are simulation values calculated directly in Python 3.9.6 and numpy 2.0.2 on a notebook (Apple M1 Max, macOS 26.5). Raw data and code: Working directory measure/robot-arm-basics/rl/'s ttenv.py(environment), baselines.py, thr.py, result out/rl/baselines.json, out/rl/throughput.json. The toy problem is intentionally kept small to demonstrate principles, and there are no actual robot experiments (hardware not measured). The initial ball states are from the DeepMind dataset (CC-BY 4.0). References were accessed on 2026-10-18. Writing rules: Measured data is the default, unmeasurable items are marked as "Not measured", and unverified items are marked as "Unverified".

Table of Contents

• 1. Intuition: Results Arrive Late
• 2. Illustration: The Toy Problem of This Series
• 3. Framework: Markov Decision Process
• 4. Measurement: How Far Can We Go Without Learning?
  • 4.1 Running Environments in Bulk
• 5. Limitations
• 6. Conclusion
• References

1. Intuition: Results Arrive Late

The methods in parts 1 to 10 calculated "what is correct" in advance. Reinforcement learning (RL) does not do that calculation. It tries, observes the outcome, and shifts actions slightly toward the direction that yielded better results. This is identical to how a child learns to throw a ball.

In table tennis, this "result" arrives late. At the exact moment you pull the racket back, you cannot know whether it was a good move; 0.5 seconds later, when the ball lands on the opponent's table, the signal of success finally arrives. In the meantime, the arm has made dozens of decisions. Attributing which decision contributed to the success(credit assignment) is the core difficulty of reinforcement learning, and the discount factor and value function in this post, along with the advantage function in part 12, are mechanisms designed to handle this problem.

Part 10 asked, "What should be left to learning?" Parts 11 to 15 revisit the learning side of that question from first principles. Part 11 establishes the framework, part 12 directly implements PPO, part 13 covers rewards and action spaces, part 14 addresses the transition from simulation to reality, and part 15 covers evaluation methods.

2. Illustration: The Toy Problem of This Series

To finish learning within a few minutes and run multiple seeds to take the median, the problem must be small. Therefore, it was constructed as follows.

• 2D Side View. Only the table tennis table's length direction (y) and height (z) are retained, while the width (x) is discarded. Coordinates are the same as in parts 1 to 10.
• 1-DOF Pendulum Racket. The axis of rotation is at (y, z) = (−1.95, 0.90) m, and an arm of length 0.75 m hangs downward, swinging back and forth. The normal of the racket face is perpendicular to the arm, so hitting the ball after passing the bottom point (φ > 0) lofts it forward and upward. Joint limits are φ ∈ [−0.9, 0.9] rad, velocity 20 rad/s, and acceleration 300 rad/s² (same magnitudes as the assumptions in part 8).
• Ball. Actual incoming shot data from the DeepMind dataset regarding y·z position, velocity, and ω_x spin toward the robot were used as-is. Flight uses the model from part 6 (gravity, drag, Magnus, bounce), and racket rebound uses the model from part 7 (e_r = 0.8, friction ignored). Physics is integrated at 1 ms intervals.
• Success. It is a success if the bounced ball crosses the net and lands on the opponent's side of the table (0 < y < 1.37 m).
• Ball Splitting. Out of 7,315 balls passing near the racket arc (among 14,703 balls heading toward the robot), 1,000 are set aside for evaluation and not used for training (part 15).

Because it is 1-DOF, the racket angle and racket velocity cannot be chosen independently. The timing of when you hit determines the racket face angle. Consequently, some balls cannot be returned no matter how you hit them, and this limitation is measured as the "upper bound" in section 4.

3. Framework: Markov Decision Process

Reinforcement learning problems are usually formulated as a Markov decision process (MDP). Mapping the five elements to the table tennis arm yields the following.

Markov Property. This is the assumption that "knowing only the current state makes the past unnecessary." If only the ball position is given and velocity is omitted, this assumption breaks down, because even at the same position, it is impossible to know whether the ball is approaching or receding. It also breaks down when there is a delay. If you are looking at the ball from 20 ms ago, you also need to know the commands you sent in the meantime to understand the present. Therefore, real systems include multiple recent observations in the state. The low-level policy in D'Ambrosio et al. 2024 takes the last 8 steps of ball position/velocity and joint states as inputs (arXiv:2408.03906 Section II-C).

Policy and Value. A policy π(a|s) is a rule for choosing actions given a state. The policy in this series is a Gaussian policy where a neural network outputs the mean and samples actions from a normal distribution. The value function is "what the expected discounted sum of future rewards will be if this policy is followed from this state onwards."

되돌림(return):     G_t = r_t + γ r_{t+1} + γ² r_{t+2} + …
상태 가치:          V(s)    = E[ G_t | s_t = s ]
행동 가치:          Q(s, a) = E[ G_t | s_t = s, a_t = a ]
이점(advantage):    A(s, a) = Q(s, a) − V(s)        「평소보다 얼마나 나은 행동이었나」

The advantage is used by PPO in part 12 to determine the direction to modify actions. You must reward **"better results than usual"** rather than just "good results" to distinguish whether the ball was easy or the action was good.

Discount Factor and Horizon. γ scales down future rewards by a factor of γ at each step. One can think that the future is "visible" for approximately 1/(1 − γ) steps ahead.

The episode lengths in the toy problem measured using a random policy had 5th, 50th, and 95th percentiles of 62, 105, and 137 steps (at 10 ms per step). Because the success reward arrives only once at the end of the episode, with γ = 0.9, that reward dwindles to 0.9⁸⁰ ≈ 0.0002 for a swing that started 80 steps prior. Therefore, γ = 0.99 is used. However, a longer horizon increases the variance of the returns, and dealing with this is the GAE of part 12.

4. Measurement: How Far Can We Go Without Learning?

Before starting learning, a baseline is measured. As will be reiterated in part 15, a learning curve without a baseline tells you nothing. These are the results for 1,000 evaluation balls.

• If the model is incorrect, planning can perform worse than a constant. A planner that chose the "best-looking" action for each ball using a spin-agnostic model (40%) performed worse than using the exact same action for all balls (46%). This is because the incorrect model drifts in different directions for different balls. The impact of part 6's statement "5 cm error without spin knowledge" is manifested here in the success rate.
• Upper Bound 85%. Even when choosing actions with foreknowledge of outcomes in the same action space, 15% of the balls could not be returned. This is the limitation of a 1-DOF racket. If learning exceeds this number, it means the action space is different (direct control is compared with hybrid in part 13).
• To spoil the outcome, PPO in part 12 reached a median of 83% using direct control on the same balls.

4.1 Running Environments in Bulk

Reinforcement learning consumes many samples. A single training run in part 12 uses about 1.4 million environment steps. Therefore, vectorizing multiple environments and running them simultaneously is fundamental. Throughput was measured by running N environments with the same array operations in numpy (excluding other computations, discarding the first of 5 runs, and taking the median).

When running a single environment, the cost of a single Python step dominates; as arrays grow larger, that cost is distributed across multiple environments. Rudin et al. 2021 pushed the same idea using a GPU physics engine, simulating thousands of robots in parallel and reporting that a flat-terrain walking policy was trained in under 4 minutes on a single workstation GPU (arXiv:2109.11978 abstract). D'Ambrosio et al. 2024 trained a base policy for about 2.4 billion steps using 6,000 parallel simulation workers (Section II-C). This is revisited in part 14.

5. Limitations

• The 2D, 1-DOF setup differs from a real table tennis arm. The left-right position of the ball, the left-right orientation of the racket face, and other components of the rotation axis were all discarded.
• The racket produces commanded accelerations without error (the control problem from part 9 is omitted). Delays and noise are introduced separately in part 14.
• Throughput values are specific to this toy environment. Throughput for physics engines like MuJoCo is unmeasured.

6. Conclusion

• The reinforcement learning framework consists of five elements: state, action, transition, reward, and discount. For the table tennis arm, these correspond to "states of the ball and arm," "joint commands or striking parameters," "physics," "whether it scored," and "how far ahead to look."
• Because delays violate the Markov property, observation history is included in the state.
• Since the success reward arrives 1 second later, a long horizon such as γ = 0.99 is required.
• Baselines are measured prior to training. In the toy problem, they are 1.5% for random, 40% for the incorrect-model planner, and 85% for the action space upper bound.

TT Briefing. You only know if you did well after the ball lands. That's why we set a large discount factor of 0.99, so praise reaches back to the swing made 1 second ago!

References

All materials were accessed on 2026-10-18.

• D'Ambrosio et al., Achieving Human Level Competitive Robot Table Tennis, arXiv:2408.03906, Section II-C (Policy Inputs).
• Rudin, Hoeller, Reist, Hutter, Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learning, arXiv:2109.11978, Abstract.
• Google DeepMind, competitive_robot_table_tennis, CC-BY 4.0.