Systematic classification differences across eye movement detection algorithms

Eye movement (EM) detection is a critical step in most eye-tracking (ET) research, typically relying on detectors-specialized algorithms designed to segment raw ET data into discrete oculomotor events. However, variability in detection algorithms and the lack of standardized evaluation frameworks hinder transparency and reproducibility across studies. In this work, we introduce pEYES, an open-source toolkit designed to streamline EM detection and enable robust, quantitative comparisons between detectors. The toolkit provides implementations for several widely used threshold-based detectors, along with multiple standardized evaluation procedures for assessing detection performance. Using pEYES, we evaluated seven detection algorithms on two publicly available human-annotated datasets containing recordings of subjects freely viewing color images. Performance was assessed using metrics such as Cohen's kappa, relative timing offset and deviation, and a sensitivity index ( $d\text{'}$ ) for fixation and saccade onsets and offsets. Engbert's adaptive velocity-threshold algorithm consistently matched or outperformed the other detectors, occasionally achieving human-level precision. In contrast, several other detectors exhibited substantial variability in performance between datasets. We also found systematic differences in detection scores between fixation and saccade boundaries, with fixation offsets and saccade onsets detected more reliably than their counterparts. These findings highlight the importance of task- and dataset-specific detector selection in EM analysis. The pEYES toolkit is freely available, and its codebase - along with the analyses presented in this report - is accessible at https://github.com/huji-hcnl/pEYES . We invite the research community to use, extend, and contribute to its ongoing development. Through open collaboration, we aim to advance the rigor and reproducibility of EM detection practices.