Convolutional neural networks can decode eye movement data: A black box approach to predicting task from eye...

Area of Science:

• Cognitive Science
• Computer Science
• Neuroscience

Background:

• Traditional methods for classifying tasks from eye movement data rely on predefined cognitive models or aggregated features.
• Black box convolutional neural networks (CNNs) can identify patterns in raw data but face challenges in generalization due to interpretability issues.

Purpose of the Study:

• To develop and validate a CNN-based approach for classifying cognitive tasks using raw eye movement data.
• To assess the performance of CNNs in distinguishing between search, memorize, and rate tasks based on visual exploration patterns.

Main Methods:

• A CNN classifier was trained and validated on two eye movement datasets (Exploratory and Confirmatory) comprising participants viewing indoor/outdoor scenes.
• Eye movement data were formatted as timelines (x-coordinate, y-coordinate, pupil size) and minimally processed images.
• Subsets of data were analyzed to determine the unique contribution of each data component (timeline vs. image, and within timeline data).

Main Results:

• Timeline data consistently yielded higher classification accuracy than image data.
• The 'Memorize' condition was frequently misclassified as 'Search' or 'Rate'.
• Pupil size provided less unique information for classification compared to x- and y-coordinates.

Conclusions:

• CNNs offer a practical and reliable "black box" solution for classifying cognitive tasks from eye movement data.
• Raw timeline data, particularly spatial coordinates, are highly informative for task classification.
• The findings support the use of deep learning models for analyzing complex visual behavior.