Prototype-based sample-weighted distillation unified framework adapted to missing modality sentiment analysis.

Area of Science:

• Artificial Intelligence
• Natural Language Processing
• Machine Learning

Background:

• Missing modality sentiment analysis presents significant real-world challenges.
• Multimodal network optimization often suffers from imbalance when modalities are absent.
• Existing research has largely overlooked optimization imbalance in missing modality scenarios.

Purpose of the Study:

• To introduce a unified framework (PSWD) for sentiment analysis that addresses missing modalities and optimization imbalance.
• To develop efficient cross-modal feature fusion and robust training strategies for incomplete multimodal data.

Main Methods:

• Utilized a transformer-based cross-modal hierarchical cyclic fusion module for feature integration.
• Implemented sample-weighted distillation to transfer knowledge effectively from complete to incomplete data.
• Introduced a prototype regularization network to balance modality gradients adaptively.

Main Results:

• The proposed PSWD framework demonstrated superior performance on benchmark datasets (IEMOCAP, MSP-IMPROV).
• Achieved state-of-the-art results compared to existing baseline methods for missing modality sentiment analysis.
• Validated the effectiveness of sample-weighted distillation and prototype regularization in handling missing data and optimization imbalance.

Conclusions:

• The PSWD framework successfully bridges sentiment analysis between missing and full modalities.
• The prototype regularization network offers a flexible, structure-agnostic approach applicable to broader multimodal research.
• The method shows significant potential for real-world sentiment analysis applications with incomplete data.