Emotion-specific modality effects in auditory and visual perception of emotion

Area of Science:

• Neuroscience
• Psychology
• Communication Studies

Background:

• Emotion perception typically occurs in dynamic multisensory contexts.
• Previous research shows multimodal gains in audiovisual recognition but lacks clarity on modality-specific contributions to affective judgments across different emotions.

Purpose of the Study:

• To investigate how emotion perception varies across audio-only, video-only, and audio-visual modalities.
• To examine if these variations depend on specific emotion categories.

Main Methods:

• An emotion perception experiment was conducted using audio-only (AO), video-only (VO), and audio-visual (AV) speech stimuli.
• Participants rated the valence and arousal of seven emotions (angry, calm, disgust, fearful, happy, sad, surprise).
• Representational similarity analysis was used to compare response patterns across modalities.

Main Results:

• Audio-only stimuli were perceived as more negative and weaker in arousal compared to visual or audio-visual stimuli.
• Response patterns for audio-visual stimuli were more similar to video-only than audio-only.
• Modality effects varied by emotion: happy and disgust showed differences in arousal and valence in AO conditions, surprise was rated more negatively in VO, while angry, fearful, and sad emotions showed no significant modality differences.

Conclusions:

• Affective judgments are not solely based on general audiovisual enhancement; the salience of sensory modalities differs across emotion categories.
• Specific unimodal cues can be as informative as multimodal information for emotion perception.
• Findings have implications for clinical assessment, social communication, and affective neuroscience research.