Reward Design and Action Space: Sparse vs. Shaped Rewards, Reward Hacking, Joint Limit Penalties, Direct Control vs. Hybrid

Status: Draft (2026-10-18), Measured. Numbers are simulation values directly calculated using Python 3.9.6 and numpy 2.0.2 on a laptop (Apple M1 Max, macOS 26.5). Raw data and code: working directory measure/robot-arm-basics/rl/, results ttenv.py, ppo.py, exp_rl.py. out/rl/rew_*, out/rl/hyb_*, out/rl/ppo_base_*All training uses 3 seeds, and evaluation is conducted on 1,000 balls not used in training. There are no real robot experiments (hardware not measured). References were accessed on 2026-10-18. Writing rules: Measured data is the default; unmeasured items are marked as "Unmeasured", and unverified items are marked as "Unverified".

Table of Contents

• 1. Intuition: Doing as Told
• 2. Five Rewards
• 3. Measurement: Rewards
  • 3.1 Reward Hacking
  • 3.2 Sparse and Shaped Rewards
  • 3.3 Joint Limit Penalties
• 4. Action Space: Direct Control vs. Hybrid
  • 4.1 What Was Read
  • 4.2 Choices in Real Systems
• 5. Limitations
• 6. Conclusion
• References

1. Intuition: Doing as Told

Reinforcement learning agents do not know the designer's intent. They only know how to maximize the sum of rewards. If you tell them "reward when hitting the ball," they will find the easiest way to hit the ball, and they do not care whether that method actually lands the ball on the table. The phenomenon where rewards exploit loopholes that deviate from the intended goal is called reward hacking. Amodei et al. (2016) cited this as one of the five concrete problems in AI safety.

The action space is a design choice of equal weight. You can have the policy determine joint accelerations every 10 ms, or you can have it decide just once "when and how hard" to hit while leaving the rest to the classical methods from Episodes 1-10. The former is flexible but requires a lot of learning; the latter learns quickly, but if the classical part is flawed, it remains trapped by those limitations.

2. Five Rewards

All use direct control (joint acceleration every 10 ms), default PPO settings (Episode 12), and 19,200 episodes.

Training uses rewards, but evaluation always uses the same criterion, which is "did it land on the opponent's table?" This is because comparing reward numbers across experiments with different reward functions is meaningless.

3. Measurement: Rewards

3.1 Reward Hacking

The "Hit Only" and "Ball Speed" rewards were huge successes in terms of their reward criteria. The racket hit 98% of the balls, and the "Ball Speed" policy sent balls flying at an average of 28.6 m/s. However, almost no balls landed on the table. Both policies swung their joints to the limit in every episode. This is because maximizing racket speed makes it easier to hit the ball and makes the ball move faster. With a real robot, this policy would slam the joint stoppers on every single stroke.

The lesson is simple: evaluate by the real goal, not the reward numbers, and suspect hacking if rewards go up while the goal remains unchanged.

3.2 Sparse and Shaped Rewards

Shaped rewards are often recommended as "early training guides," but in this problem, the sparse reward (86.8%) actually slightly outperformed the shaped reward (82.9%) (the difference is within range given 3 seeds). Two reasons are suspected. First, since even a random policy succeeds 1.5% of the time (Episode 11), one or two out of every 64 episodes receive a success signal. When success is not extremely rare, guidance is less necessary. Second, intermediate rewards like "0.3 for hitting" teach the model that "hitting and missing the table" is better than "missing entirely," which mixes in directions slightly different from the true goal. Shaped rewards pay off in problems where success is much rarer (e.g., smashing with a 7-DOF arm from the start), and even then, they must be designed not to conflict with the goal (unmeasured).

3.3 Joint Limit Penalties

With a single penalty, episodes hitting the limits dropped from 4–79% down to 5–8%, without sacrificing success rate (89.2%). The default policy of Episode 12 hit the limits not because it needed to, but because it was never told not to. Some argue that safety conditions are more reliably handled as constraints rather than rewards. Real robots apply hard limits on the controller side separately from penalties.

4. Action Space: Direct Control vs. Hybrid

Direct controlis identical to Episode 11. There are around 100 decisions per episode, and the policy must figure out the timing, trajectory, and racket angle entirely on its own.

Hybridselects only two numbers at the first step: a correction to the contact angle Δφ (±0.15 rad) and the racket angular velocity at impact φ̇_c (0–8 rad/s). The rest is handled by classical methods. Using a ball flight model (Episode 6) to predict where and when the ball enters the racket arc, a quintic trajectory (Episode 8) is generated to reach that spot at that speed and executed as is. As a reinforcement learning problem, this becomes a single-step problem (contextual bandit). Training was conducted for 120 iterations with 64 balls per iteration (7,680 balls total).

Two predictors were tested: a predictor that ignores spin, like the "B model" in Episode 6, and a predictor that accounts for spin.

Baselines without training were the best constant action in the hybrid action space at 46%, the grid planner with the spin-ignorant model at 40%, and the upper bound of the grid chosen using true outcomes at 85% (Episode 11).

4.1 What Was Read

• Hybrid is fast. Direct control required 5,120 balls (seed median) to exceed 40%, whereas hybrid took 1,280 to 1,920 balls. Because evaluations were performed every 640 balls, this is a coarse measurement. This is because it only needs to learn two numbers. Tebbe et al. (2021) also noted in their abstract that treating only the racket state (orientation and speed) at the moment of impact as the action achieved accurate results within 200 episodes without pre-training.
• However, it is trapped by the limitations of the classical part. Hybrid using a spin-ignorant predictor could not exceed the best constant action (46%). If the predicted contact point is wrong, a correction of Δφ ±0.15 rad is not even enough to touch the ball. When the predictor was made aware of spin, performance rose to 66%. The performance of the hybrid approach is determined by the quality of the classical component. This is similar to Episode 10's statement: "Learning is cheap only when equations narrow down what is unknown."
• Hybrid is safe. Because the trajectory is planned within limits, episodes hitting joint limits were under 1%. Direct control reached up to 79% without penalties.
• Hybrid training fluctuates. The evaluation success rate of the spin-aware hybrid oscillated between 10% and 81% during training, and taking the final value yielded 39% for one seed. 64 balls per iteration is a noisy sample for a bandit problem. The rule of reporting the "final" rather than the "best-ever" moment was strictly followed (Episode 15).
• The reason direct control reached close to the hybrid upper bound (85%)is presumed to be because direct control can combine striking angle and speed more flexibly within 1 DOF (e.g., decelerating mid-swing) (estimate).

4.2 Choices in Real Systems

Going down the list delegates more to classical components (trajectory, inverse kinematics, ball prediction). For the first open-hardware robot, starting with a hybrid approach to gain sample efficiency and safety is recommended, followed by identifying weaknesses in the classical part (such as spin prediction) and then fixing them or expanding them through learning.

5. Limitations

• Only a single set of coefficients for shaped rewards (0.3, 0.5, 0.05) was tested. Results may vary with different coefficients (unmeasured).
• The hybrid policy input consists only of the initial observation, and predicted contact point information was not included in the inputs. Including it could yield better performance (unmeasured).
• Due to using only 3 seeds, differences between similar setups (sparse vs. shaped, default vs. penalty) are not statistically distinguishable.

6. Conclusion

• Rewards are the sole channel to communicate what is desired. Rewarding "hitting" or "hitting hard" yields a policy that hits 98% of the time and scores 0%. Always evaluate using the true goal.
• In this problem, sparse rewards alone were sufficient. Shaped rewards are a tool for when successes are extremely rare.
• Safety is maintained only when explicitly stated. A single joint limit penalty reduced limit collisions tenfold.
• The hybrid approach, which learns only strike parameters, is fast and safe, but is constrained by flaws in the classical component. Switching to a spin-aware predictor raised success from 46% to 66%.

A word from TT. "When we said 'praise the robot for hitting the ball,' the robot hit every ball and scored zero! Praise should only be given to what you truly want!"

References

All materials were accessed on 2026-10-18.

• Amodei et al., Concrete Problems in AI Safety, arXiv:1606.06565, Abstract.
• Tebbe, Krauch, Gao, Zell, Sample-efficient Reinforcement Learning in Robotic Table Tennis, arXiv:2011.03275(ICRA 2021), Abstract.
• D'Ambrosio et al., arXiv:2309.03315(Comparison of Action Spaces), arXiv:2408.03906(Skill Policies and High-Level Controllers).
• Büchler et al., arXiv:2006.05935 v1.