A Resource-Rational Account of Human Eye Movements During Immersive Visual Search

Area of Science:

• Cognitive Science
• Computational Neuroscience
• Artificial Intelligence

Background:

• Visual search behavior is extensively studied, but computational models struggle with complex virtual reality (VR) environments.
• Existing ideal observer models have limitations due to strong assumptions about world structure.

Purpose of the Study:

• To model immersive visual search in VR using reinforcement learning (RL).
• To test if RL agents can predict naturalistic search behavior and align with human gaze data.

Main Methods:

• Visual search was framed as a reinforcement learning problem using a convolutional neural network.
• RL agents learned policies over latent states (meta-Markov decision process) to allocate attention.
• Learned policies were compared to human gaze data from a VR visual search experiment.

Main Results:

• Trained RL agents converged to an ideal-observer model for simple stimuli.
• The learned resource-rational policy showed qualitative and quantitative alignment with human gaze data in VR.
• Model performance and human correspondence depended on representational features.

Conclusions:

• Naturalistic visual search behavior can be partially explained by resource-rational allocation of cognitive resources.
• The choice of representational features significantly influences the alignment between computational models and human behavior.