Weber-Fechner law in temporal difference learning derived from control as inference.

Area of Science:

• Computational Neuroscience
• Machine Learning
• Artificial Intelligence

Background:

• Standard reinforcement learning (RL) uses linear temporal difference (TD) error updates, treating all rewards equally.
• Biological systems exhibit nonlinearities in TD errors, leading to optimistic or pessimistic learning biases.
• These nonlinear biases are potentially adaptive features of biological learning.

Purpose of the Study:

• To explore a theoretical framework for leveraging nonlinearity between update degree and TD errors in RL.
• To investigate the applicability of the Weber-Fechner law (WFL) within a control-as-inference framework for RL.
• To demonstrate the practical utilities of WFL in RL through a reward-punishment system.

Main Methods:

• Analysis of a control-as-inference framework to identify nonlinear relationships in RL.
• Derivation and application of the Weber-Fechner law (WFL) to model the relationship between TD errors and update magnitudes.
• Implementation of a reward-punishment framework to numerically demonstrate WFL's effects on RL policies.

Main Results:

• The Weber-Fechner law (WFL) was identified, describing how perception of TD error changes with value function intensity.
• WFL implementation demonstrated accelerated escape from low-reward situations.
• WFL implementation showed enhanced pursuit of minimal punishment.

Conclusions:

• The proposed RL algorithm incorporating WFL accelerates reward maximization and effectively suppresses punishments.
• Nonlinear update rules, inspired by biological learning, offer significant advantages in RL.
• WFL provides a viable mechanism for introducing beneficial biases into artificial learning systems.