How Reward and the Elimination of Dependence Thereon Lead Evolution to High-Bandwidth Learning1

Area of Science:

• Artificial Intelligence
• Computational Neuroscience
• Machine Learning

Background:

• Current AI learning methods, particularly Reinforcement Learning (RL), suffer from low efficiency compared to biological learners, attributed to the 'Cost of Generality' and a 'Reward Bottleneck'.
• Biological intelligence efficiently processes rich sensory information, often independent of explicit reward signals, suggesting alternative learning mechanisms.
• Existing AI struggles to replicate this efficiency due to reliance on sparse, scalar reward feedback, limiting learning bandwidth.

Purpose of the Study:

• To propose and computationally explore High-Bandwidth Learning (HBL) as a biologically plausible pathway to overcome the Reward Bottleneck in AI.
• To investigate if reward-driven learning can act as a catalyst for the evolution of HBL, enabling AI to learn from diverse information streams.
• To demonstrate the scalability and effectiveness of evolved HBL agents in both simple and complex, high-dimensional tasks.

Main Methods:

• A population-based evolutionary approach was used, modeling reward-driven learning with RL (A2C) and incorporating a neuromodulatory connection weight update mechanism for integrating non-reward information.
• Experiments were conducted on a minimalistic 2D navigation task to assess initial learning speed and efficiency improvements.
• Scalability was tested on a high-dimensional quadruped locomotion task with randomized physical parameters.

Main Results:

• Evolved HBL agents demonstrated a 300-fold increase in learning speed on the navigation task compared to pure RL agents.
• Agents successfully eliminated dependence on reward, learning exclusively from non-reward information via local neuromodulation.
• In the quadruped task, evolved HBL agents significantly outperformed RL baselines in both learning speed and final performance, driven by reward-agnostic neuromodulation.

Conclusions:

• Reward-driven learning can serve as an evolutionary precursor to High-Bandwidth Learning, facilitating the gradual integration of non-reward information.
• HBL offers a promising, biologically inspired framework for developing more efficient, autonomous, and general artificial intelligence.
• The findings have implications for understanding the Baldwin effect, the evolution of autonomy, and the generality of human intelligence.