Table Tennis Arm Reinforcement Learning: Past the Baseline, but Most Gains Were Not from Learning

Status: Draft (2026-10-18), measured. All success rates are values inside the simulator, while camera period, delay, noise, and spin hint accuracy are assumed values. Training was conducted on 2026-10-18 as a temporary pod on a single GPU node of a home lab Kubernetes cluster (a single RTX 4060 for laptops). The pod limits were 6 CPU cores and 4 GiB of memory, sharing resources with other services on the same node. The software stack consists of MuJoCo 3.16.0, MuJoCo Warp (warp-lang 1.18.0), Python 3.11, and PPO (hidden layer 256). The physics matches parts 2 and 5 (time step 0.5 ms, racket stiffness 4e6). Evaluation was performed using 500 evaluation balls separated from the training balls (seed 0 samples identical to part 5's baselines) under DeepMind's default unrandomized physics. RL values are the median of 3 seeds, with parentheses showing the seed min–max. The baseline and nominal planner are deterministic and were run once. Raw data (training logs, evaluation results, recorded trajectories, Warp comparison) is in the working directory measure/sim-tt-rl/ .

Table of Contents

• 1. Intuition: What to Learn
• 2. Experimental Design
• 3. Results
  • 3.1 Reading Honestly: Where Did the Gains Come From?
  • 3.2 RL Learning Joints Directly
  • 3.3 The Same Ball, Three Methods
  • 3.4 Remaining Failures
• 4. Why a GPU Was Needed: MuJoCo Warp
• 5. What Happened During Execution
• 6. Unmeasured Items
• 7. To the Next Part
• References

1. Intuition: What to Learn

There are broadly two ways to teach table tennis using reinforcement learning. One is to make the robot's joints move directly . The policy outputs joint targets at every moment and figures out how to hit the ball on its own. The other is to let classical control handle what it does best, and only learn the decisions . Inverse kinematics, trajectories, and joint control are kept as-is, and the policy only chooses striking parameters such as "when, at what angle, and how hard to hit."

Part 13 of "Basics of Table Tennis Robot Arms" (Reward and Action Space, ../robot-arm-basics/13-reward-and-action-space.md) compared these two using a pendulum racket and showed that hybrid learns quickly, but gets stuck if the predictor does not know the spin. This time, we asked the same question using part 5's full-scale robot (2-axis gantry + 2-axis wrist) and the public ball distribution.

2. Experimental Design

• Robot limits. Unlike part 5, hardware planning limits (Part 3) were actually enforced: horizontal axis 3 m/s and 40 m/s², height axis 3 m/s and 30 m/s², and wrist 20 and 30 rad/s.
• Domain randomization (during training only). Table restitution 0.80–0.98, friction 0.1–0.3, drag coefficient 0.40–0.55, camera delay 0–60 ms, and observation noise 0–5 mm. Magnus effect was fixed to DeepMind values as it is paired with the spin in the public dataset (Part 2).
• Spin type hint. A variation incorporating the conclusion of Part 6 as-is. One of topspin, backspin, or no spin is provided, and the average spin of that type is used as a prior for trajectory matching. During evaluation, it was assumed that the hint is correct only 80% of the time(perturbed to 70–100% during training).
• Three evaluation conditions. True state (knowing the ball's state directly), 120 Hz with 30 ms delay, and 120 Hz with 100 ms delay (camera noise σ 3 mm). All are assumed values.

3. Results

(Trained with 0–120 ms delay, without randomization; the CPU version is a single seed.)

3.1 Reading Honestly: Where Did the Gains Come From?

• Baseline → Nominal: +21.4–27.0%p, no learning. The baseline gave up on 115 balls during the planning stage in the true state (74 due to joint limits, 37 because the ball did not arrive at a reachable location, and 4 due to no predicted bounce). The nominal planner swung even by clipping out-of-limit solutions, reducing planning failures to 0 and returning 36 of those 115 balls. "Hit but out of bounds" also dropped from 68 to 10. Both are differences originating from the planner.
• Nominal → RL (a): +1.6–4.6%p. The net gain added by learning is small. The median landing error decreased from 14.6 to 10.8 cm (true state), showing a slight improvement in aiming.
• Spin hint: +10.4%p at 100 ms delay (58.2 → 68.6%). However, nominal + hint without learning was also 66.6%. This gain also comes from using the hint as a prior for trajectory matching, not from learning. With short delays (30 ms or less), there was no gain from the hint.
• The conclusion points in the same direction as the pendulum experiments in Part 13 and the results of "Spin Type Hint + Trajectory Matching" in Part 6. A good planner and good prediction (priors) come first, and learning is fine-tuning on top of that.

3.2 RL Learning Joints Directly

• Good at hitting, weak at aiming. The number of missed balls in the true state was 40, fewer than the hybrid model (64), but "hit but out of bounds" (45) and "hit into own court" (37) were much higher than the hybrid model (10 and 2, respectively). The median landing error also exceeded 27.1 cm, more than double that of the hybrid model.
• Vulnerable to delay. A policy trained with delay randomized to 0–60 ms dropped to 29.6% at 100 ms, with large variance across seeds (12.8–38.6%). Expanding the training delay to 0–120 ms raised it to 52.0% at 100 ms. It cannot handle delays not seen during training.
• Collapses without randomization. A policy trained without randomization achieved 72.6% under the training condition (120 Hz / 30 ms), but dropped to 2.0% in the true state and 0.0% at 100 ms. It fails if observations differ even slightly from training conditions.

3.3 The Same Ball, Three Methods

Evaluation ball #21 is redrawn using recorded trajectories (5 ms intervals) under the 120 Hz / 30 ms condition. Clicking the image plays it back 3x slower.

The baseline attempts planning at 0.2 s, gives up because it is outside joint limits, and does not move the racket. Both the nominal planner and the post-training hybrid return the same ball to the opponent's court. The fact that the two trajectories are nearly identical echoes the discussion in Section 3.1.

3.4 Remaining Failures

• Workspace. All 64 balls missed by the post-training hybrid in the true state were cases where solutions were clipped due to joint limits. The remaining failures are issues of the robot's reachable range, not learning.
• Low balls. Out of 43 balls where the ball height at the striking surface was less than 10 cm, almost none were successfully received by any method (0% for baseline, 9% for nominal, 12% for hybrid). This provides another rationale for Part 3's proposal to prioritize the depth (y) axis with a two-stage axis design.
• 100 ms delay. Failures tripled even in the hybrid method. Reducing camera delay remains the single biggest leverage point (same conclusion as Parts 3 and 5).

4. Why a GPU Was Needed: MuJoCo Warp

Direct joint RL requires many samples. The CPU version (6 processes) trained 17.5 million decisions over 30 minutes but remained stuck at a 0% success rate. Moving the same environment to MuJoCo Warp allowed running 4,096 worlds in parallel on a single GPU.

(Physics steps are MuJoCo steps with dt 0.5 ms. Success rates during training are measured on the training distribution mixed with randomization and exploration noise, differing from the evaluation values in Section 3. The variation in hybrid training time across seeds is due to sharing the CPU with other tasks on the same node.)

We first verified whether physics remained identical when moving to Warp.

• Sending 200 identical balls in CPU and Warp, the median position difference after 1 second was 0.003 cm.
• In tunneling tests with racket plate thicknesses of 6 and 14 mm and ball speeds of 10–40 m/s, the Warp version recorded 0 penetrations across all 300 trials (150 per thickness), and the coefficient of restitution matched the CPU version up to the fourth decimal place.
• Comparing the CPU and Warp environments 64 times with identical policy inputs yielded identical results in all 64 runs.

5. What Happened During Execution

The training pod was launched using kubectl run --rm -i . The intention was to have the pod automatically delete itself when the container finished. However, the attach connection of the first pod dropped due to a TLS error after about 4 hours, and at that exact moment, --rm deleted the training pod. After that, the container printed a line every minute to keep the connection alive, and the results were exported externally every ten minutes. We set a maximum execution time (activeDeadlineSeconds) on the pod as a final safety measure.

6. Unmeasured items

• Before running on real hardware. All results are within the simulator. We do not know if the same order (nominal > RL fine-tuning) will appear on a real robot.
• Actual camera latency. 30 ms and 100 ms are assumptions. We have not yet measured the actual latency distribution of our camera.
• Spin type classification accuracy. 80% of the evaluation is an assumption. We have not measured how accurately we can actually identify spin types using the ball, sound, and posture.
• The actual restitution and friction of the racket rubber, and the robot's limit values (velocity, acceleration, force) are also assumptions.
• RL (b) was trained with 0–120 ms latency, no randomization, and the CPU version uses 1 seed.

7. To the next part

The next step is to rerun the same evaluation by adding low balls and the depth axis to widen the workspace into the robot model. The story of aligning physics and the referee with real footage is in Part 8.

TT's note. We were excited that reinforcement learning significantly beat the baseline, but it turned out that most of it was thanks to the new planner without learning. You have to separate and measure the "learned share" and the "design-changed share" to see the true skill!

References

All links were verified as accessible when accessed on 2026-10-18.

• MuJoCo: https://github.com/google-deepmind/mujoco, MuJoCo Warp: https://github.com/google-deepmind/mujoco_warp
• Initial ball state upon release: https://github.com/google-deepmind/competitive_robot_table_tennis(© 2025 Google LLC, CC BY 4.0. Data was not redistributed, only simulation results are included.)
• Further reading: Part 5 Classical Control Baseline, Part 6 DeepMind Table Tennis Robot Improvements, "Basics of Table Tennis Robot Arms" Part 13 Reward and Action Space(../robot-arm-basics/13-reward-and-action-space.md)
• Quoted code and figure trajectories are outputs from TT Lab experiment code.