Audiovisual synchrony perception for speech and music assessed using a temporal order judgment task

Area of Science:

• Auditory Perception
• Audiovisual Integration
• Human Sensitivity

Background:

• Audiovisual asynchrony perception is crucial for multisensory integration.
• Previous research primarily focused on continuous speech streams, leaving music perception less explored.
• Understanding sensitivity to temporal discrepancies is key to audiovisual processing.

Purpose of the Study:

• To investigate human sensitivity to audiovisual asynchrony in briefly presented speech and music.
• To compare sensitivity to temporal discrepancies between speech and musical stimuli.
• To provide empirical data on audiovisual asynchrony perception in music.

Main Methods:

• Presentation of speech (letters, syllables) and music (guitar, piano) video clips at various stimulus onset asynchronies (SOAs).
• Participants performed unspeeded temporal order judgments (TOJs) to determine perceived order of auditory and visual streams.
• Analysis of accuracy using just noticeable difference (JND) and point of subjective simultaneity (PSS).

Main Results:

• Significantly better TOJ accuracy (smaller JND) for speech compared to music (guitar, piano).
• Visual lead required for piano music PSS; auditory lead typical for guitar music PSS.
• Speech PSS varied with specific speech sounds; speech asynchrony sensitivity exceeded prior estimates.

Conclusions:

• Humans exhibit greater sensitivity to audiovisual asynchrony in speech than in music.
• This study offers the first empirical evidence on audiovisual asynchrony perception for musical stimuli.
• Sensitivity to speech asynchrony is higher than previously suggested, especially with brief stimuli.