Cross-modal BERT model for enhanced multimodal sentiment analysis in psychological social networks

Human emotions in psychological social networks often involve complex interactions across multiple modalities. Information derived from various channels can synergistically complement one another, leading to a more nuanced depiction of an individual's emotional landscape. Multimodal sentiment analysis emerges as a potent tool to process this diverse array of content, facilitating efficient amalgamation of emotions and quantification of emotional intensity.

This paper proposes a cross-modal BERT model and a cross-modal psychological-emotional fusion (CPEF) model for sentiment analysis, integrating visual, audio, and textual modalities. The model initially processes images and audio through dedicated sub-networks for feature extraction and reduction. These features are then passed through the Masked Multimodal Attention (MMA) module, which amalgamates image and audio features via self-attention, yielding a bimodal attention matrix. Subsequently, textual information is fed into the MMA module, undergoing feature extraction through a pre-trained BERT model. The textual information is then fused with the bimodal attention matrix via the pre-trained BERT model, facilitating emotional fusion across modalities.

The experimental results on the CMU-MOSEI dataset showcase the effectiveness of the proposed CPEF model, outperforming comparative models, achieving an impressive accuracy rate of 83.9% and F1 Score of 84.1%, notably improving the quantification of negative, neutral, and positive affective energy.

Such advancements contribute to the precise detection of mental health status and the cultivation of a positive and sustainable social network environment.