Leif Doering, Daniel Schmidt, Moritz Melcher, Sebastian Kassing, Benedikt Wille, Tilman Aach, Simon Weissmann
This paper provides a convergence proof for Proximal Policy Optimization (PPO) by interpreting its update scheme as approximate policy gradient ascent and addresses an issue in Generalized Advantage Estimation (GAE).
Proximal Policy Optimization (PPO) is a popular algorithm used in deep reinforcement learning, but its theoretical underpinnings have been incomplete. This study presents a way to understand PPO's policy update process as an approximation of policy gradient ascent, which helps explain why PPO works well in practice. The authors also discovered a problem with how PPO estimates advantages, particularly at the end of episodes, and propose a correction that improves performance in certain environments. Their findings contribute to a better theoretical understanding of PPO and suggest practical improvements for its implementation.
