Sensory modality affects emotional dimension perception and frontoparietal brain responses

The ability to perceive other people's emotions is essential for social interaction and communication. Emotion perception involves complex cognitive processes triggered by visual, auditory, or both stimuli. However, it remains unclear how people use auditory, visual, and combined audiovisual cues to evaluate emotional valence and arousal. To address this, we conducted an emotion perception task using functional near-infrared spectroscopy to measure frontoparietal brain responses. Seven emotion categories (Angry, Calm, Disgust, Fear, Happy, Sad, and Surprise) portrayed by actors were presented under three modality conditions: audio-only (AO), audiovisual (AV), and video-only (VO). We observed significant differences between AO, AV, and VO in the behavioral evaluation of emotional valence and arousal. In addition, AV showed significant activations for emotion categories in the frontoparietal network, but not for AO or VO. Moreover, brain-behavior relationships showed that higher oxygenated hemoglobin (HbO) activities were associated with more positive valence ratings in the VO condition, but with stronger arousal ratings in the AV condition. These findings reveal that sensory modality affects brain activity (HbO) and behavior evaluation (valence and arousal ratings) in the perception of emotional dimensions.