A comprehensive model of audiovisual perception: both percept and temporal dynamics

Area of Science:

• Neuroscience
• Cognitive Science
• Computational Modeling

Background:

• The brain processes sparse sensory input to interpret the environment, such as locating audiovisual stimuli.
• Multisensory integration is an ill-posed inverse problem, requiring joint information processing and constraints to resolve uncertainty.
• Perception models have typically focused on either the outcome (percept) or the temporal dynamics, not both.

Purpose of the Study:

• To extend a previously developed audiovisual perception model to predict both the percept and its temporal dynamics.
• To capture the phenomenon of audiovisual perception comprehensively by integrating both dimensions.
• To investigate how contextual conditions influence multisensory information processing and perception.

Main Methods:

• A data-driven approach using behavioral analysis to develop a Bayesian network model.
• The model infers different percepts and the temporal dynamics of the audiovisual perception process.
• Context-specific independence analyses were employed to explore contextual effects on information handling.

Main Results:

• The developed model successfully predicts both the percept and temporal dynamics of audiovisual perception.
• Contextual analyses revealed that similar percepts from unisensory or non-fused multisensory stimuli arise from different underlying processes, evidenced by distinct temporal dynamics.
• The model demonstrates the influence of both bottom-up (stimulus-driven) and top-down (instruction-induced) factors on perception.

Conclusions:

• A unified model can capture both the outcome and temporal dynamics of audiovisual perception.
• Context plays a crucial role in modulating how the brain processes multisensory information, affecting temporal dynamics even when percepts are similar.
• The findings highlight the interplay between stimulus properties and cognitive control in shaping perceptual experiences.