Time Stand Still: Effects of Temporal Window Selection on Eye Tracking Analysis

Area of Science:

• Cognitive Science
• Psycholinguistics
• Computational Neuroscience

Background:

• Reproducibility in research is challenged by researcher degrees of freedom.
• Time window selection in time series analysis is a critical, yet often unexamined, source of variability.
• Eye tracking studies, particularly in the visual world paradigm, generate rich time series data susceptible to windowing effects.

Purpose of the Study:

• To investigate the impact of time window selection on statistical results in time series data analysis.
• To demonstrate how varying time windows can alter parameter estimates and p-values in a logistic linear mixed effects model.
• To highlight the need for transparency and awareness regarding the influence of time window choices on experimental outcomes.

Main Methods:

• Utilized data from a visual world eye tracking experiment.
• Examined 8281 unique time windows by systematically varying start times and lengths.
• Applied a consistent logistic linear mixed effects model across all selected time windows, controlling for time, age, noise, and word frequency.

Main Results:

• Substantial variations in parameter estimates and p-values were observed across different time windows.
• Changes in time window selection, even within narrow ranges (100-200 ms), led to reversals in the direction of parameter estimates (positive to negative).
• The choice of time window significantly influenced the interpretation of statistical significance and effect sizes.

Conclusions:

• Time window selection is a crucial factor that can substantially bias research findings and impact reproducibility.
• Advocates for transparency in reporting time window selection methods.
• Recommends preregistration and multiverse model exploration as strategies to mitigate bias and enhance the reliability of results.