Object-scene semantics correlation analysis for image emotion classification.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Psychology

Background:

• Image emotion classification (IEC) predicts human emotions from images, a field challenged by the 'affective gap' between visual pixels and high-level emotional perception.
• Existing IEC methods often struggle due to limitations in semantic representation, hindering accurate emotion prediction.

Purpose of the Study:

• To address the affective gap in IEC by proposing a novel network, the Object-Scene Attention Fusion Network (OSAFN).
• To enhance emotion classification by integrating object-level concepts and scene-level reasoning as auxiliary information.

Main Methods:

• OSAFN utilizes external tools for concept extraction and Appraisal-based Chain-of-Thought (Appraisal-CoT) prompting for scene information generation.
• Attention-based modules align semantic and visual features, while an adaptive fusion strategy integrates object and scene streams.
• A polarity-aware contrastive loss is employed to model emotion hierarchies and improve fine-grained category discrimination.

Main Results:

• Numerical experiments on four affective datasets demonstrated the superior performance of OSAFN.
• The proposed method effectively leverages object and scene information for enhanced visual representations and emotion classification.

Conclusions:

• OSAFN represents a notable contribution to the field of image emotion classification.
• The fusion of object and scene semantic information effectively mitigates the affective gap, leading to improved model performance.