Related Experiment Video
Updated: Jul 17, 2026
11:18
Closed-loop Neuro-robotic Experiments to Test Computational Properties of Neuronal Networks
Published on: March 2, 2015
Exploration-Exploitation Mechanisms in Recurrent Neural Networks and Human Learners in Restless Bandit Problems.
D Tuzsus1, A Brands1, I Pappas2
1Department of Psychology, Biological Psychology, University of Cologne, Cologne, Germany.
Computational Brain & Behavior
|July 16, 2026
Summary
Recurrent neural network models (RNNs) show human-level performance in decision-making tasks, but differ in exploration strategies. Humans explore based on uncertainty, unlike RNNs, highlighting distinct cognitive and computational mechanisms.
Area of Science:
- Cognitive Neuroscience
- Computational Neuroscience
- Machine Learning
Background:
- Decision-making involves balancing exploration (seeking new information) and exploitation (choosing known rewards).
- Recurrent neural network (RNN) models are increasingly used in reinforcement learning research for their meta-learning capabilities.
- Restless bandit tasks are standard for studying exploration-exploitation trade-offs.
Purpose of the Study:
- To compare the performance of various RNN architectures against human learners on restless bandit problems.
- To investigate computational mechanisms underlying exploration and exploitation in both humans and RNNs.
- To identify similarities and differences in decision-making strategies between artificial and biological learning systems.
Main Methods:
- Comprehensive comparison of multiple RNN architectures, including LSTM networks with computational noise, against human participants.
- Behavioral data analysis to identify signatures of perseveration and directed exploration.
- Analysis of RNN hidden unit dynamics to understand neural correlates of choice behavior.
Main Results:
- The best-performing RNN architecture (LSTM with noise) achieved human-level performance on the task.
- Both humans and RNNs exhibited higher-order perseveration, with a more pronounced effect in RNNs.
- Human learners, unlike RNNs, demonstrated directed exploration influenced by uncertainty.
- RNNs showed exploratory choices linked to disrupted predictive signals in low-value states, akin to a win-stay-loose-shift strategy.
Conclusions:
- Meta-learning RNNs can replicate human-level performance in complex decision-making tasks.
- Significant differences exist in exploration strategies, particularly the role of uncertainty in human decision-making.
- RNN hidden unit dynamics offer insights into neural mechanisms of exploration, aligning with findings in primate prefrontal cortex.
Related Concept Videos
01:05Instinctive Drift
Instinctive drift refers to the tendency of animals to revert to their innate behaviors despite repeated reinforcement. Breland and Breland demonstrated this concept in an experiment with a raccoon. The raccoon was trained to pick up two coins and place them in a container in exchange for food. Initially, the raccoon learned to associate the coins with food, making them a conditioned stimulus or a substitute for food. However, over time, the raccoon became less willing to put the coins into the...
01:14Avoidance Learning and Learned Helplessness
Avoidance learning and learned helplessness are critical concepts in understanding behavioral responses to negative stimuli.
Avoidance learning occurs when an organism learns that a specific behavior can prevent an unpleasant outcome. For example, a student who receives a bad grade may start studying harder to avoid future poor grades. This behavior persists even when the negative outcome is no longer present. Avoidance learning is powerful because it maintains behavior in the absence of the...
Avoidance learning occurs when an organism learns that a specific behavior can prevent an unpleasant outcome. For example, a student who receives a bad grade may start studying harder to avoid future poor grades. This behavior persists even when the negative outcome is no longer present. Avoidance learning is powerful because it maintains behavior in the absence of the...
01:12Observational Learning
Albert Bandura's observational learning, also known as imitation or modeling, occurs when a person observes and imitates another's behavior. It is a quicker process than operant conditioning. A well-known example is the Bobo doll study, where children who saw an adult acting aggressively towards the doll were more likely to act aggressively when left alone, compared to those who observed a nonaggressive adult. Many psychologists view observational learning as a form of latent learning because...