Improved IEC performance via emotional stimuli-aware captioning.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Psychology

Background:

• Image emotion classification (IEC) aims to identify emotions in images.
• Current methods struggle with the affective gap between visual features and high-level emotions.
• Existing techniques often rely on semantic information, which has limitations.

Purpose of the Study:

• To enhance image emotion classification by integrating visual features with auxiliary information from image captions.
• To address the affective gap in IEC by leveraging natural language processing advancements.
• To develop a novel network for emotion-aware image captioning and classification.

Main Methods:

• Introduced the emotional stimuli-aware captioning network (ESCNet) for augmented visual representations.
• Developed an affective captioning dataset for pre-training emotion-related caption generation.
• Utilized a fusion module with cross-attention and self-attention to correlate image and caption features.
• Implemented a variable-weight loss function to focus on challenging samples.

Main Results:

• The proposed ESCNet approach demonstrated superior performance compared to state-of-the-art models on multiple public datasets.
• Ablation studies and visualization confirmed the effectiveness of the network's components.
• The integration of generative captions significantly improved emotional discernment.

Conclusions:

• The study successfully bridges the affective gap in IEC by combining visual and textual information.
• ESCNet offers a promising direction for more accurate and robust image emotion classification.
• The findings highlight the potential of leveraging NLP techniques in computer vision tasks.