Unifying Top-down and Bottom-up Scanpath Prediction Using Transformers

Area of Science:

• Computer Science
• Cognitive Science
• Neuroscience

Background:

• Existing visual attention models typically focus on either top-down or bottom-up attention control.
• These models are often evaluated using distinct visual search and free-viewing tasks.

Purpose of the Study:

• To introduce a unified model, the Human Attention Transformer (HAT), capable of predicting both top-down and bottom-up attention control.
• To establish a new state-of-the-art in computational models of human visual attention.

Main Methods:

• Developed a novel transformer-based architecture incorporating a simplified foveated retina for spatio-temporal awareness.
• Implemented a sequential dense prediction architecture outputting dense heatmaps for each fixation, avoiding discretization issues.

Main Results:

• HAT achieves state-of-the-art performance in predicting human fixation scanpaths across visual search (target-present, target-absent) and free-viewing tasks.
• The model enhances the interpretability of human gaze behavior.
• Demonstrated effectiveness, generality, and interpretability in computational attention.

Conclusions:

• HAT represents a significant advancement in computational attention, offering a unified approach to modeling visual attention.
• The model's success and interpretability are expected to drive future research in attention prediction.
• HAT provides a robust framework for understanding human behavior in attention-demanding scenarios.