Stimuli-Aware Visual Emotion Analysis.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Psychology

Background:

• Visual emotion analysis (VEA) is increasingly important for understanding social media content.
• Existing VEA methods often use general features, overlooking specific emotional stimuli.
• Human emotion perception is complex and ambiguous, posing challenges for AI.

Purpose of the Study:

• To develop a stimuli-aware visual emotion analysis (VEA) method inspired by the Stimuli-Organism-Response (S-O-R) model.
• To integrate stimuli selection directly into an end-to-end VEA network.
• To enhance the accuracy and interpretability of visual emotion recognition.

Main Methods:

• Proposed a three-stage VEA method: stimuli selection (S), feature extraction (O), and emotion prediction (R).
• Utilized off-the-shelf tools for selecting emotional stimuli (color, object, face) from images.
• Designed Global-Net, Semantic-Net, and Expression-Net for distinct feature extraction and a hierarchical cross-entropy loss for refined prediction.

Main Results:

• The proposed stimuli-aware VEA method achieved superior performance compared to state-of-the-art approaches on four public datasets.
• Ablation studies confirmed the effectiveness of individual components of the method.
• Visualizations demonstrated the interpretability and validity of the approach.

Conclusions:

• The stimuli-aware method represents a significant advancement in visual emotion analysis.
• Integrating stimuli selection enhances VEA by focusing on relevant image components.
• The method offers improved accuracy and interpretability for emotion recognition in images.