Adaptive bias-variance trade-off in advantage estimator for actor-critic algorithms

Area of Science:

• Reinforcement Learning
• Robotics
• Machine Learning

Background:

• Actor-critic methods are prominent in continuous control tasks.
• Advantage estimators, crucial critics in actor-critic, blend state values and sample returns.
• Balancing bias and variance in estimators is key to reducing errors, but optimal combinations fluctuate during training.

Purpose of the Study:

• To explore the relationship between bias/variance indicators and their optimal combination in advantage estimation.
• To develop a general form of adaptive advantage estimators for reduced estimation errors.
• To evaluate the performance of proposed adaptive estimators against existing methods.

Main Methods:

• Analysis of bias and variance sources using indicators from previous work on adaptive advantage estimation (AAE).
• Numerical experiments to investigate the relationship between indicators and optimal combination strategies.
• Development of a general adaptive combination form for state values and sample returns.

Main Results:

• Established a general form for adaptive combinations of state values and sample returns.
• Demonstrated that these adaptive estimators achieve low estimation errors.
• Achieved similar or superior performance compared to Generalized Advantage Estimators (GAE) on simulated robotic locomotion tasks.

Conclusions:

• Adaptive advantage estimation offers a flexible approach to optimize actor-critic performance.
• The proposed method effectively manages the bias-variance trade-off throughout training.
• This work provides a foundation for more robust and efficient continuous control algorithms.