In the Eye of Transformer: Global-Local Correlation for Egocentric Gaze Estimation and Beyond

Area of Science:

• Computer Vision
• Artificial Intelligence
• Human-Computer Interaction

Background:

• Egocentric gaze estimation is crucial for understanding human intentions in daily activities.
• Existing methods often struggle to integrate global scene context with local visual cues.

Purpose of the Study:

• To develop the first transformer-based model for egocentric gaze estimation.
• To enhance the accuracy of predicting human gaze from first-person video perspectives.

Main Methods:

• A novel transformer encoder embeds global scene context as an additional visual token.
• A global-local correlation module explicitly models the interplay between global and local visual information.
• Validation on EGTEA Gaze+ and Ego4D datasets.

Main Results:

• The proposed model significantly outperforms previous state-of-the-art methods.
• Ablation studies confirm the effectiveness of the global-local correlation module.
• The model demonstrates strong generalization capabilities on related tasks like gaze saccade/fixation prediction and action recognition.

Conclusions:

• Global-local correlation is a key representation for accurate egocentric gaze prediction.
• The transformer-based approach offers a powerful new direction for gaze estimation research.
• The model's performance highlights its potential for real-world applications requiring human intention understanding.