Predicting Goal-directed Human Attention Using Inverse Reinforcement Learning.

Area of Science:

• Computer Vision
• Cognitive Science
• Human-Computer Interaction

Background:

• Human gaze behavior prediction is crucial for both behavioral and computer vision.
• Existing models primarily predict free-viewing behavior and struggle with goal-directed visual search tasks.
• Predicting visual search, where individuals look for specific objects, remains a challenge.

Purpose of the Study:

• To develop the first inverse reinforcement learning (IRL) model for predicting human visual search behavior.
• To model internal reward functions and policies guiding human visual search.
• To understand and predict scanpath behavior during target object identification.

Main Methods:

• Proposed an inverse reinforcement learning (IRL) model to learn human reward functions and policies during visual search.
• Modeled viewer's internal belief states as dynamic contextual belief maps of object locations.
• Created COCO-Search18, the largest dataset for goal-directed visual search, comprising 300,000 fixations from 10 participants across 18 target categories in 6202 images.

Main Results:

• The IRL model significantly outperformed baseline models in predicting human search scanpaths on the COCO-Search18 dataset.
• The model demonstrated superior accuracy in terms of similarity to human search behavior and search efficiency.
• Recovered reward maps from the IRL model highlighted target-dependent object prioritization patterns, suggesting learned object context.

Conclusions:

• The proposed IRL model offers a novel and effective approach to predicting goal-directed human gaze behavior.
• The model's ability to learn internal reward functions provides insights into the cognitive processes underlying visual search.
• The findings contribute to advancing computer vision applications by enabling more human-like gaze prediction in complex search tasks.