Cue integration in categorical tasks: insights from audio-visual speech perception

Area of Science:

• Cognitive Science
• Neuroscience
• Speech Perception

Background:

• Previous cue integration models focused on continuous dimensions, assuming weights proportional to sensory reliability.
• These models may not apply to categorical perception, like phoneme identification, where environmental variance is also crucial.

Purpose of the Study:

• To investigate cue integration in a categorical audio-visual speech perception task.
• To determine if human performance aligns with a Bayes-optimal observer model for categorical dimensions.

Main Methods:

• Computational and experimental investigation of audio-visual phonemic labeling.
• Assessing sensitivity to sensory uncertainty in auditory and visual cues on a trial-by-trial basis.

Main Results:

• Human performance in audio-visual phonemic labeling is consistent with a Bayes-optimal observer.
• Participants are sensitive to sensory uncertainty, and environmental variability within categories influences cue weighting.
• Visual sensory variability is better estimated from multi-cue than single-cue performance.

Conclusions:

• Human cue integration in categorical tasks is more complex than previously modeled, incorporating environmental variability.
• Findings provide a foundation for understanding the mechanisms of categorical cue integration in speech perception.