Deep learning and eye tracking: Convolutional neural networks provide converging evidence for experience-driven...

Eye tracking during visual search generates spatiotemporally rich but complex data. Traditional analyses often utilize simplified metrics (saccade landings, dwell time, etc.) that necessarily exclude a substantial fraction of the variance in the raw eye data. Here, we asked if deep learning might aid scientists in objectively incorporating such discarded data into analyses. Convolutional neural networks (CNNs) are supervised machine learning tools that excel at classifying biological data. We built several CNNs that learn from raw eye-position time-course data to classify the location of relevant stimuli (e.g., search targets/distractors). We train each CNN on two-thirds of the data and cross-validate on the rest, comparing classification accuracy to chance via traditional frequentist testing and hierarchical Bayesian modeling. Using data from two of our previous visual search studies (Massa et al., Atten Percept Psychophys 86(4):1108-1119, 2024; Grubb & Li, Atten Percept Psychophys 80:822-828, 2018), CNNs successfully classified the location of distractors with a "history as a sought target," finding evidence for reflexive, experience-driven overt attention within each oculomotor dataset. Successful prediction of distractor location generalized to a third dataset without additional training (Doyle et al., Atten Percept Psychophys 87:721-727, 2025) and outperformed a traditional saccade-landing metric. Feature visualization illustrated how the CNNs learn from eye-position samples near distractors early in trials and opposite distractors later in trials, suggestive of reflexive attentional allocations towards distractors followed by corrective shifts in gaze. We thus validate our CNN-based approach and highlight its utility in analyzing the spatiotemporally rich data gathered from eye tracking during visual search.