Emotion-specific modality effects in auditory and visual perception of emotion

Emotion is typically perceived in dynamic multisensory contexts. Although previous studies have demonstrated multimodal gains in audiovisual recognition (i.e., general enhancement in audiovisual conditions), they have not clarified how visual and auditory information contribute to affective judgments or whether their contribution differs by emotion categories. The current study investigated how emotion perception varies across modalities and examined whether these variations depend on emotion categories. We conducted an emotion perception experiment using audio-only (AO), video-only (VO), and audio-visual (AV) speech stimuli. Participants evaluated the valence (positive/negative) and arousal (weak/strong) of stimuli depicting seven emotions (angry, calm, disgust, fearful, happy, sad, and surprise). Overall, AO stimuli tended to be perceived as more negative in valence and weaker in arousal compared to VO or AV stimuli. In representational similarity analysis, the response patterns for AV were more similar to VO than to AO. Happy and disgust stimuli were rated as weaker in arousal and less positive or less negative in valence in AO condition than in other conditions. For surprise, VO stimuli were rated more negatively than AO and AV. However, angry, fearful, and sad emotions did not differ between modalities. These findings indicate that affective judgments are not solely driven by general enhancement from combined audiovisual cues; rather, the relative importance of sensory modalities varies by emotion, suggesting that specific unimodal cues can be as informative as multimodal information. These modality-specific contributions to emotion perception have implications for clinical assessment, social communication, and affective neuroscience.