Visual and Auditory Components in the Perception of Asynchronous Audiovisual Speech

Area of Science:

• Auditory and Visual Neuroscience
• Cognitive Psychology
• Computational Modeling

Background:

• Audiovisual speech perception research investigates how auditory and visual speech signals are integrated.
• Probabilistic models are used to understand the sensory and decisional processes in synchrony judgments.
• Two models, independent-channels and Bayesian, have been proposed to explain these processes.

Purpose of the Study:

• To compare the performance of independent-channels and Bayesian models in explaining audiovisual speech synchrony judgments.
• To determine how manipulations in visual speech quality affect sensory and decisional processes.
• To assess the interpretability and goodness-of-fit of each model.

Main Methods:

• Fitted independent-channels and Bayesian probabilistic models to synchrony judgment data.
• Data involved audiovisual speech stimuli with varying visual component quality and audiovisual offsets.
• Assessed model fit using goodness-of-fit statistics and parameter identifiability.

Main Results:

• The Bayesian model failed to fit asymmetric data and showed poor goodness-of-fit for half the observers.
• The independent-channels model successfully accommodated asymmetric data and had good fit for most observers.
• The independent-channels model provided interpretable parameters revealing effects of visual quality on sensory and decisional processes.

Conclusions:

• The independent-channels model provides a more robust and interpretable account of audiovisual speech synchrony perception.
• Visual speech quality significantly influences both sensory encoding and decision-making in perceiving synchrony.
• Model comparison is crucial for advancing our understanding of multisensory integration mechanisms.