Recurrent Neural Network Exploration Strategies During Reinforcement Learning Depend on Network Capacity

Area of Science:

• Computational neuroscience
• Artificial intelligence
• Cognitive science

Background:

• Artificial neural networks (ANNs) model biological brains for complex task understanding.
• Recurrent neural networks (RNNs) with 48 hidden units achieve human-level performance in bandit tasks but use different strategies.

Purpose of the Study:

• To systematically investigate how network capacity (number of hidden units) influences computational mechanisms and performance in ANNs.
• To compare RNN behavior across different capacities and with human learners.

Main Methods:

• Computational modeling of RNNs with varying capacities (e.g., 48 vs. 576 hidden units).
• Utilized a restless multi-armed bandit task common in neuroscience research.
• Compared network exploration strategies (directed vs. random) and learning parameters (switch rate, perseveration) with human learners.

Main Results:

• High-capacity RNNs exhibited increased directed exploration and decreased random exploration compared to low-capacity networks.
• RNNs with 576 hidden units showed exploration strategies closer to human learners.
• Despite closer strategies, RNNs still deviated from humans in switch rate and perseveration.

Conclusions:

• Network capacity is crucial for shaping exploration strategies in reinforcement learning.
• Human learners may employ resource-rational strategies, allocating more cognitive resources to task solving.
• Findings contribute to developing more human-like ANNs for insights into brain mechanisms.