Timing Is Everything: Temporal Dynamics of BOLD Responses to Naturalistic Features in Movie Watching

Naturalistic stimuli, such as movies and narratives, are increasingly used in cognitive neuroscience to map cognitive and affective processes onto brain activity measured with functional MRI (fMRI). Features extracted from movies span multiple levels, from computational descriptions of visual and auditory input to physiological signals and subjective ratings. However, the temporal alignment between these features and the blood-oxygen-level-dependent (BOLD) response can vary considerably, and the commonly used canonical hemodynamic response function (HRF) with temporal derivatives may not adequately capture these delays. In this study, we analyzed three movie-watching datasets and examined a range of features, including visual luminance and contrast, auditory pitch, a physiological measure of pupil size, and subjective ratings of theory of mind. We performed cross-correlation analyses between feature time series and fMRI responses across the whole brain. Our results show that the canonical HRF captures sensory features well but misaligns with slower signals such as pupil size and subjective ratings. Notably, raw pupil size and subjective ratings exhibited stronger correspondence with delayed hemodynamic responses. Lastly, we showed that video data can be used as continuous input to measure hemodynamic response functions across brain regions. These findings highlight the critical role of temporal alignment in mapping naturalistic features to brain activity. Accounting for feature-specific delays may improve the interpretability and accuracy of fMRI studies using complex, real-world stimuli.