How Are Audiovisual Simultaneity Judgments Affected by Multisensory Complexity and Speech Specificity?

Area of Science:

• Auditory perception
• Visual perception
• Human-computer interaction

Background:

• Stimulus complexity influences the temporal integration window (TIW).
• Quantitative evaluation of ecologically valid stimuli on TIW is lacking.
• Audiovisual complexity is linked to the correspondence between auditory onset and visual movement.

Purpose of the Study:

• To quantitatively evaluate the effect of audiovisual complexity on TIW using ecologically valid stimuli.
• To investigate the role of speech-specific factors in temporal integration.
• To determine if TIW width is influenced by stimulus complexity and/or speech specificity.

Main Methods:

• Experiment 1: Simultaneity judgment task with piano, shakuhachi, and speech video clips.
• Experiment 2: Comparison of natural speech sounds versus white noise stimuli.
• Regression analysis to assess the relationship between complexity and TIW width.

Main Results:

• TIW was wider for speech compared to piano and shakuhachi stimuli.
• TIW width was dependent on the degree of audiovisual complexity.
• Natural speech stimuli resulted in a wider TIW than white noise.

Conclusions:

• Audiovisual stimulus complexity, particularly the obviousness of auditory-visual correspondence, affects TIW width.
• Speech-specific factors contribute to a broader TIW.
• Both complexity and speech specificity play a role in shaping temporal integration in audiovisual perception.