VisdaNet: Visual Distillation and Attention Network for Multimodal Sentiment Classification.

Area of Science:

• Natural Language Processing
• Computer Vision
• Artificial Intelligence

Background:

• Social media sentiment analysis is crucial for decision-making.
• Users increasingly employ multimodal content (text and images) for expression.
• Effective multimodal fusion is essential for accurate sentiment classification.

Purpose of the Study:

• To propose VisdaNet, a novel multimodal sentiment classification approach.
• To enhance information integration from short text and associated images.
• To improve the quality of modal information and text-image interaction modeling.

Main Methods:

• Developed a knowledge augmentation module integrating image captions and short text.
• Implemented CLIP-based knowledge distillation for noise reduction in multimodal fusion.
• Introduced CLIP-based visual aspect attention for single-text multi-image scenarios.

Main Results:

• VisdaNet achieved state-of-the-art (SOTA) performance on the Yelp multimodal dataset.
• Ablation experiments confirmed the effectiveness of individual components within VisdaNet.
• The model demonstrated superior capability in modeling text-image interactions.

Conclusions:

• VisdaNet offers a significant advancement in multimodal sentiment classification.
• The proposed methods effectively address challenges in multimodal information fusion.
• This approach enhances the accuracy and robustness of opinion analysis systems.