Feature fusion based transformer for sentiment analysis in social networks

Area of Science:

• Artificial Intelligence
• Natural Language Processing
• Computer Vision

Background:

• Social media analysis for mental health evaluation is challenging due to diverse data types.
• Single-modal sentiment analysis techniques struggle with complex, multimodal social media information.
• Existing methods may yield inaccurate or contradictory user emotional state assessments.

Purpose of the Study:

• To propose an effective multimodal sentiment analysis framework for evaluating user mental health.
• To overcome the limitations of traditional single-modal approaches in social media analysis.
• To enhance the accuracy of emotional state detection from social media content.

Main Methods:

• Utilized RoBERTa for text feature extraction and ResNet50 for image feature extraction.
• Employed a multimodal Transformer architecture for feature alignment and fusion.
• Integrated a fully connected network (FCN) for final sentiment classification.

Main Results:

• The proposed Feature Fusion Based Transformer (FFBT) model demonstrated superior performance.
• FFBT achieved a 4.1% increase in accuracy compared to existing sentiment analysis algorithms.
• FFBT showed a 5% improvement in F1-scores on a custom social media dataset.

Conclusions:

• The FFBT framework effectively fuses multimodal features for accurate sentiment analysis.
• Multimodal sentiment analysis is crucial for reliable mental health evaluation from social media.
• FFBT offers a promising advancement in computational mental health research.