Visual-Auditory Integration and High-Variability Speech Can Facilitate Mandarin Chinese Tone Identification

Area of Science:

• Auditory perception
• Speech processing
• Cross-modal integration

Background:

• Tone identification is crucial for spoken language comprehension.
• Visual information depicting pitch contours can aid auditory tone perception (visual effect).
• The influence of prior tonal language experience on this visual effect is not fully understood.

Purpose of the Study:

• Investigate the visual effect in combined visual-auditory integration.
• Examine the role of speech variability (high vs. low) in this effect.
• Determine if tonal language learning experience modulates the visual effect's strength.

Main Methods:

• Participants: Mandarin-naïve listeners, Mandarin second language learners, and native Mandarin speakers.
• Task: Tone identification (rising vs. falling pitch) using auditory tones and visual pitch contour displays (moving arrows).
• Design: Priming paradigm with four multimodal conditions: auditory-visual minus (A-V-), visual-only (A-V+), auditory-only (A+V-), and combined (A+V+).

Main Results:

• Mandarin-naïve listeners showed a visual effect in accuracy (d-prime) with cross-modal integration, but not in response time.
• The visual effect in accuracy and response time under unimodal visual-only conditions was significant only for high-variability speech.
• Less prior tonal language experience correlated with a stronger visual effect across all listener groups.

Conclusions:

• Visual-auditory integration offers advantages but also disadvantages for the visual effect in tone perception.
• High-variability speech and visual-auditory integration jointly facilitate tone perception through speech symbolization and categorization.