Actions in the Eye: Dynamic Gaze Datasets and Learnt Saliency Models for Visual Recognition.

Area of Science:

• Computer Vision
• Human-Computer Interaction
• Neuroscience

Background:

• Current computer vision systems use bag-of-words models from sparse interest points for object and action recognition.
• While computer vision and human visual processing share similarities (saccade and fixate regimes), their methodologies remain distinct.
• Bridging this gap requires integrating human visual behavior data into computer vision models.

Purpose of the Study:

• To create the first large-scale, publicly available human eye-tracking dataset for dynamic computer vision tasks.
• To analyze the stability and patterns of human visual search during action and scene recognition.
• To develop and evaluate end-to-end trainable computer vision systems that leverage human eye movement data.

Main Methods:

• Collected human eye movements from 19 subjects viewing 497,107 frames of dynamic video stimuli under task-controlled and free-viewing conditions.
• Developed novel dynamic consistency and alignment measures to analyze visual search patterns.
• Built and trained end-to-end computer vision systems using predicted human fixations and advanced computer vision techniques.

Main Results:

• The study presents a unique, large-scale dataset for video action recognition, crucial for computer vision research.
• Analysis revealed remarkable stability in human visual search patterns across subjects.
• Human eye movements (fixations) were accurately predicted and demonstrated to improve computer vision system performance.

Conclusions:

• Human eye-tracking data offers valuable insights for advancing computer vision, particularly in action and scene recognition.
• The developed dataset and methods facilitate the creation of more biologically plausible and effective computer vision systems.
• Leveraging human fixation predictions in end-to-end systems achieves state-of-the-art recognition results, highlighting the potential of human-computer synergy.