Computational Counterfactuals Reveal Non-Additive Audiovisual Semantics in Natural Movie Responses

Area of Science:

• Neuroscience
• Cognitive Science
• Computational Neuroscience

Background:

• Natural perception integrates auditory and visual information.
• Decomposing audiovisual stimuli may not fully capture neural processing.
• Existing models often focus on unimodal or additive processing.

Purpose of the Study:

• To investigate if intact audiovisual movie perception is better predicted by native semantics than unimodal reconstructions.
• To introduce a computational-counterfactual framework for analyzing audiovisual processing.
• To identify brain regions and networks involved in coherent audiovisual semantic emergence.

Main Methods:

• Utilized 7 Tesla movie fMRI data from 176 participants.
• Employed a computational-counterfactual framework comparing native audiovisual models with additive unimodal baselines.
• Applied content-aware purged cross-validation, representational similarity analysis, feature replacement, and content gating.

Main Results:

• The native audiovisual model significantly outperformed the additive unimodal baseline.
• Strongest prediction gains were observed in auditory, visual, and dorsal attention systems.
• Analyses indicated that the advantage stemmed from semantic emergence and network-specific routing, not just sensory discrepancy.

Conclusions:

• Intact movie viewing elicits cortical responses aligned with native audiovisual meaning.
• The brain processes coherent audiovisual semantics beyond additive unimodal information.
• This suggests a distinct neural mechanism for holistic audiovisual perception.