Multimodal sentiment analysis: hybrid classification model with image and text feature descriptors.

Area of Science:

• Artificial Intelligence
• Computer Science
• Natural Language Processing

Background:

• Understanding human emotions via text and images is crucial for applications like content personalization and social media analysis.
• Traditional sentiment analysis often uses single modalities, missing complementary information from other sources.
• Human-Computer Interaction (HCI) benefits from advanced emotion recognition.

Purpose of the Study:

• To propose a novel multimodal sentiment analysis framework integrating text and image data.
• To enhance sentiment analysis accuracy by leveraging complementary information from multiple modalities.
• To develop an optimized hybrid model for improved emotion understanding.

Main Methods:

• Text preprocessing involved tokenization, stopword removal, and stemming, extracting features like N-grams, emojis, and TF-IDF.
• Image preprocessing used object detection, deriving multitexton and Shape Local Binary Texture (SLBT) features.
• A hybrid model combined an optimized Deep Maxout for text and a Modified Sigmoid-based Bidirectional Gated Recurrent Unit (MS-Bi-GRU) with transfer learning for images, optimized by the Innovative Beluga Whale Optimization Algorithm (IBwOA).

Main Results:

• The proposed multimodal framework demonstrated superior performance compared to conventional single-modality models.
• Experimental results showed significant improvements across various performance metrics in sentiment analysis.
• The hybrid model effectively integrated text and image features for robust emotion detection.

Conclusions:

• The novel multimodal sentiment analysis framework offers a significant advancement in understanding human emotions.
• Integrating text and image data through the proposed hybrid model enhances sentiment analysis accuracy.
• The approach shows promise for various applications requiring sophisticated emotion recognition.