Skip to yearly menu bar Skip to main content


Poster

Log-normality and Skewness of Estimated State/Action Values in Reinforcement Learning

Liangpeng Zhang · Ke Tang · Xin Yao

Pacific Ballroom #200

Keywords: [ Reinforcement Learning and Planning ] [ Hardness of Learning and Approximations ]


Abstract:

Under/overestimation of state/action values are harmful for reinforcement learning agents. In this paper, we show that a state/action value estimated using the Bellman equation can be decomposed to a weighted sum of path-wise values that follow log-normal distributions. Since log-normal distributions are skewed, the distribution of estimated state/action values can also be skewed, leading to an imbalanced likelihood of under/overestimation. The degree of such imbalance can vary greatly among actions and policies within a single problem instance, making the agent prone to select actions/policies that have inferior expected return and higher likelihood of overestimation. We present a comprehensive analysis to such skewness, examine its factors and impacts through both theoretical and empirical results, and discuss the possible ways to reduce its undesirable effects.

Live content is unavailable. Log in and register to view live content