Robust inter-subject audiovisual decoding in functional magnetic resonance imaging using high-dimensional regression

Area of Science:

• Neuroscience
• Machine Learning
• Signal Processing

Background:

• Neural decoding aims to reconstruct mental content from neuroimaging data.
• Generalizability of current decoding models across subjects and content remains unclear.
• Need for robust models that can decode perceived features without prior content-specific training.

Purpose of the Study:

• To develop generalizable neural decoding models for reconstructing perceived audiovisual features from fMRI data.
• To validate these models across different subjects, movies, and scanning environments.
• To assess the potential of these models in scientific and diagnostic applications.

Main Methods:

• Applied kernel ridge regression with temporal optimization to fMRI data acquired during film viewing.
• Generated standardized brain models for features: sound loudness, speech presence, perceived motion, face ratio, lightness, and brightness.
• Tested prediction accuracy on data from different subjects watching new movies, often in a different scanner.

Main Results:

• Significant correlations (Q_FDR<0.05) found for loudness, speech, and motion models across all 9 test movies (R¯≈0.60).
• High reproducibility of predictors across subjects was observed.
• Face ratio model showed significant correlations in 7/8 movies (R¯=0.56); lightness and brightness models lacked robustness.

Conclusions:

• The developed loudness, speech, motion, and face ratio models demonstrate validity and generalizability for complex audiovisual stimuli.
• Loudness reconstruction can differentiate musical experience, highlighting diagnostic potential.
• The approach shows promise for basic science and clinical contexts, though further validation with controlled stimuli is recommended.