Tasks Reflected in the Eyes: Egocentric Gaze-Aware Visual Task Type Recognition in Virtual Reality

Area of Science:

• Human-Computer Interaction
• Computer Vision
• Virtual Reality

Background:

• Eye tracking in AR/VR headsets enables adaptive content display by recognizing user visual tasks.
• Existing visual task recognition methods are often scene-specific, limiting their applicability in diverse environments like museums.
• There is a need for scene-agnostic approaches to generalize task recognition across various scenarios.

Purpose of the Study:

• To propose four scene-agnostic task types for broader applicability in visual task recognition.
• To develop and evaluate an egocentric gaze-aware method for task type recognition.
• To create a new dataset for training and validating scene-agnostic task recognition models.

Main Methods:

• Collected a new dataset of eye and head movement data from 20 participants across 15 VR videos.
• Defined four scene-agnostic task types to facilitate generalized recognition.
• Developed and implemented the egocentric gaze-aware task type recognition method (TRCLP).

Main Results:

• The proposed TRCLP method achieved promising results in recognizing visual task types.
• The new dataset supports research in scene-agnostic visual task recognition in VR.
• Demonstrated practical applications of the task recognition method with three examples.

Conclusions:

• Scene-agnostic task type recognition is feasible and beneficial for intelligent AR/VR applications.
• The TRCLP method and the accompanying dataset advance the field of gaze-based interaction.
• Findings provide valuable insights for content developers creating task-aware intelligent applications.