Sensory Modality Shapes the Representational Structure of Emotion Recognition

Emotion recognition is a fundamental aspect of social communication, yet most research has focused on visual emotion expressions, leaving it less understood how auditory and audiovisual cues shape perceptual organization. The present study used representational similarity analysis (RSA) to examine how the structure of emotion recognition differs across modalities.

Participants completed an emotion recognition task involving 14 emotions presented in auditory, visual, and audiovisual formats. For each modality, a confusion matrix was constructed from the distribution of emotion labels participants chose for each target emotion. To separate confusion structure from overall accuracy, the primary analyses used error-only profiles (the off-diagonal responses, excluding correct answers), which were converted into representational dissimilarity matrices (RDMs). Structure was characterized with hierarchical clustering and multidimensional scaling (MDS) of the same RDM, with the number of clusters selected by silhouette and with reliability and stability assessed by resampling.

Recognition accuracy was lower for auditory than for visual and audiovisual presentations, which did not differ. Audiovisual and visual structures were strongly correlated and organized by valence, whereas the auditory structure diverged from both and showed no valence organization. Audiovisual structure corresponded most closely to a modality-general reference, followed by visual and then auditory. In the auditory modality, emotions of similar arousal but opposite valence were more confusable. These findings indicate that audiovisual and visual emotion perception share a largely valence-based organization, whereas auditory perception is organized along partly different lines, underscoring the value of examining the representational structure of emotion recognition beyond accuracy.