DROP: Distributional and Regular Optimism and Pessimism for Reinforcement Learning.

Area of Science:

• Neuroscience and Artificial Intelligence
• Computational Neuroscience

Background:

• Temporal difference (TD) error in reinforcement learning (RL) is linked to dopamine neuron activity.
• Dopamine neurons exhibit optimistic or pessimistic responses to TD error, suggesting distributional RL.
• Existing heuristic models lack theoretical grounding for asymmetric learning rates.

Purpose of the Study:

• Introduce a novel, theoretically grounded RL model incorporating optimism and pessimism.
• Explain biological data on dopamine neuron responses to TD error.
• Develop an algorithm that leverages distributional value functions for policy improvement.

Main Methods:

• Derived a new model from control as inference principles.
• Utilized ensemble learning to estimate a distributional value function (critic).
• Improved policy (actor) based on the critic's central value.
• Proposed algorithm named DROP (distributional and regular optimism and pessimism).

Main Results:

• DROP demonstrated excellent performance and high generality across dynamic tasks.
• The heuristic model showed poor learning performance compared to DROP.
• DROP achieved learning performance comparable to state-of-the-art RL algorithms.

Conclusions:

• DROP is a theoretically grounded RL algorithm that effectively utilizes optimism and pessimism.
• The model provides a potential explanation for observed dopamine neuron behavior.
• DROP elicits the potential contributions of optimism and pessimism in RL, offering a new avenue for research.