Policy Gradients and PPO: Why We Clip Ratios, Advantage Functions and GAE, Value Functions, Entropy

Status: Draft (2026-10-18), measured. The numbers are simulation values calculated directly using Python 3.9.6 and numpy 2.0.2 on a laptop (Apple M1 Max, 10-core, macOS 26.5). Each training run used a single process (1 numpy thread), and running 8 trainings simultaneously took 5 to 7 minutes per run. Raw data and code: Working directory measure/robot-arm-basics/rl/'s ppo.py, ttenv.py, exp_rl.py, results out/rl/ppo_*_s{0,1,2}.json. There are no actual robot experiments (hardware not measured). References were accessed on 2026-10-18. Writing rules: Measured data is the default, unmeasured items are marked as "Unmeasured", and unverified items are marked as "Unverified".

Table of Contents

• 1. Intuition: A little bit in the right direction, but not too much
• 2. Figure: Clipped Objective Function
• 3. Calculations
  • 3.1 Policy Gradient
  • 3.2 Advantage Estimation and GAE
  • 3.3 PPO Loss
  • 3.4 Implementation
• 4. Measurements
  • 4.1 Learning Curves
  • 4.2 What We Read
  • 4.3 What Real Table Tennis Robots Used
• 5. Limitations
• 6. Conclusion
• References

1. Intuition: A little bit in the right direction, but not too much

Suppose you hit a ball 100 times. Some swings went in better than usual, and some did worse. Next time, you just need to swing similarly to the good ones and differently from the bad ones. That is the entirety of policy gradients. It increases the probabilitythat the policy will choose that action, proportional to how much better the result was than usual.

The problem is "how much." Experience from 100 trials is noisy. If you trust a single well-executed swing too much and alter the policy drastically, the new policy will venture into regions where previous experience provides no guidance and will collapse there. PPO (Proximal Policy Optimization) introduced a mechanism inside the objective function that stops pushing when the new policy gets too far from the old policy . Thanks to this, we can perform multiple updates using the same experience. The abstract of Schulman et al. 2017 also lists the ability to perform multiple epochs of minibatch updates as an advantage of the new objective function.

2. Figure: Clipped Objective Function

• The horizontal axis represents r, the ratio of the probabilities that the new and old policies choose the same action. If r = 1, the policy has not changed.
• If the advantage A is positive (an action better than usual), increasing r increases the objective, but once it exceeds 1 + ε, it flattens out, removing any reason to increase it further.
• Conversely, if A is negative, r is decreased, but it is not pushed below 1 − ε.
• In the flat region, the gradient is 0, so those samples contribute nothing to the update. In our measurements, about 5% of the samples were in this state in the later stages of training.

3. Calculations

3.1 Policy Gradient

Differentiating the policy performance J with respect to the policy parameters θ yields the following form.

∇J(θ) = E[ ∇ log π_θ(a | s) · A(s, a) ]

For a Gaussian policy (mean μ_θ(s), standard deviation σ), the derivative of the log probability is straightforward.

log π(a|s) = −½ ((a − μ)/σ)² − log σ − ½ log 2π
∂/∂μ       = (a − μ) / σ²
∂/∂ log σ  = ((a − μ)/σ)² − 1

If an action with a value greater than the mean was good (A > 0, a > μ), we shift the mean toward it; if an action far from the mean was good, we increase σ to explore more broadly.

3.2 Advantage Estimation and GAE

Since we cannot precisely know the advantage A = Q − V, we estimate it. By separately learning a value function V, we can compute the one-step temporal difference (TD) error.

δ_t = r_t + γ V(s_{t+1}) − V(s_t)
GAE:  Â_t = δ_t + (γλ) δ_{t+1} + (γλ)² δ_{t+2} + …
λ = 0:  Â_t = δ_t                      한 걸음만 보고 나머지는 V 를 믿는다(분산 작음, V 가 틀리면 치우침)
λ = 1:  Â_t = G_t − V(s_t)             끝까지 실제 보상을 더한다(치우침 없음, 분산 큼)

Generalized Advantage Estimation (GAE) uses λ to choose between them. The abstract of Schulman et al. 2015 explains that it trades off a slight increase in bias for a large reduction in variance using the value function. In problems like table tennis where rewards arrive 100 steps later, setting λ = 0 is dangerous because it essentially rehashes the garbage V from early training 100 times. We measure this in Section 4.

3.3 PPO Loss

r_t(θ)  = π_θ(a_t | s_t) / π_old(a_t | s_t)
L_clip  = E[ min( r_t Â_t ,  clip(r_t, 1 − ε, 1 + ε) Â_t ) ]       (키운다)
L_value = E[ (V(s_t) − (Â_t + V_old(s_t)))² ]                      (줄인다)
L_ent   = E[ 정책의 엔트로피 ]                                       (c_ent 만큼 키운다)

The entropy bonus prevents σ from shrinking too quickly and halting exploration.

3.4 Implementation

We wrote backpropagation by hand in numpy without automatic differentiation. The key point is that the gradient of the clipped objective flows only for samples where "min selects the unclipped term."

ratio = np.exp(logp_new - logp_old)
unc, clp = ratio * A, np.clip(ratio, 1 - eps, 1 + eps) * A
w = np.where(unc <= clp, A * ratio, 0.0)          # 잘린 표본은 기울기 0
z = (a - mu) / std
d_mu      = -(w[:, None] * z / std) / B            # 손실(−L_clip)의 평균에 대한 기울기
d_log_std = -(w[:, None] * (z**2 - 1)).sum(0) / B - c_ent

The configuration is as follows. The policy and value neural networks each have hidden layers of 64-64 (tanh), a learning rate of 3×10⁻⁴ (Adam), collect 64 episodes using 64 environments per iteration, update with 10 epochs and a minibatch size of 512, ε = 0.2, γ = 0.99, λ = 0.95, gradient clipping at norm 0.5, and initial σ = 0.61. Running 300 iterations yields 19,200 episodes and approximately 1.4 million environment steps. Evaluation is performed every 20 iterations using 1,000 evaluation balls, using only the mean action without probabilities.

4. Measurements

4.1 Learning Curves

The baselines were 1.5% for random, 40% for a classical planner unaware of spin, and 85% for the upper bound of the hybrid action space (Part 11).

4.2 What We Read

• Clipping saves training. Running 10 epochs on the same data without clipping caused the policy to drift away from the old policy by a KL divergence of 9 to 25 per update (whereas default PPO stays around 0.01). One seed climbed to 36% during the first 60 iterations and then collapsed, while two seeds hovered near 0% from the start. This shows that reusing the same data multiple times is inherently dangerous, and clipping prevents that danger.
• λ = 0 failed on this problem. Since rewards arrive only at the end of an episode, the one-step TD error is nearly zero, and the garbage value function from early training dictates all the signals. λ = 1 reached 84% on one seed, but stayed around 43% on the other two. Because variance is high, it relies heavily on seed luck. The median λ = 0.95 exceeded 77% across all three seeds.
• The entropy bonus was beneficial for this problem. The median rose from 83% to 88%, and the range across seeds dropped from 14%p to 0.3%p. Under the default setting where σ shrank to 0.1, one seed stopped exploring early (77%). However, because there were only 3 seeds, it is difficult to generalize this difference (Part 15).
• The learned policy hit joint limits. In the final evaluation of the default setting, episodes hitting joint limits at least once ranged from 4% to 79% depending on the seed. This was because the reward function did not penalize limits. We try adding a penalty in Part 13.
• The success rate on 1,000 training balls for the same policy matched the evaluation set within 1%p (e.g., 84.1% vs 82.9%). Because the ball distributions were identical, there was virtually no overfitting. Part 15 covers evaluation methods.

4.3 What Real Table Tennis Robots Used

Büchler et al. (arXiv 2006.05935 v1) trained a pneumatic muscle arm using PPO at 100 Hz (Part 10). Conversely, D'Ambrosio et al. 2024 trained low-level skills using black-box optimization from the evolution strategies family (BGS) instead of PPO, citing as their reason that the motions were smoother than those produced by PPO or SAC (Section II-C ofarXiv:2408.03906 ). This is the same concern as our default PPO hitting joint limits and producing jerky policies. You choose not based on "which algorithm is better" but on "what you want" (success rate, smoothness, sample efficiency).

5. Limitations

• 3 seeds, 300 training iterations. Two seeds in the default setting were still climbing at 300 iterations. Results from running longer are unmeasured.
• Common implementation tricks like observation normalization, learning rate decay, and value loss clipping were not included. Only a few values were compared for the hyperparameter settings (ε, λ, entropy coefficient).
• The experimental table in the main text of the original PPO paper (comparison by ε value, MuJoCo setup) could not be verified because the PDF could not be parsed as text (unverified). Our ε = 0.2 was chosen as a commonly used value.

6. Conclusion

• Policy gradients increase the probability of "actions that performed better than usual." Measuring what counts as "usual" is the value function, and the difference is the advantage.
• GAE's λ trades off bias and variance. In table tennis where rewards are delayed, λ = 0 failed and 0.95 was the most stable.
• PPO clips ratios to allow reusing the same data multiple times without collapsing. Without clipping, the KL divergence of a single update spiked to about 1,000–2,500 times that of the default setting and collapsed.
• On the toy pendulum racket, PPO reached 83–88% with 19k balls. A classical planner unaware of spin achieved 40% on the same balls. However, that policy aggressively hit joint limits.

Titi's Note: If you get too excited about one well-hit swing and change everything drastically, you'll miss all the subsequent balls! PPO establishes the rule: "Change only a little!"

References

All materials were accessed on 2026-10-18.

• Schulman, Wolski, Dhariwal, Radford, Klimov, Proximal Policy Optimization Algorithms, arXiv:1707.06347, abstract.
• Schulman, Moritz, Levine, Jordan, Abbeel, High-Dimensional Continuous Control Using Generalized Advantage Estimation, arXiv:1506.02438, abstract.
• Büchler et al., arXiv:2006.05935 v1.
• D'Ambrosio et al., arXiv:2408.03906, Section II-C (Learning Algorithm).