A vision-language model for multitask classification of memes.

Area of Science:

• Artificial Intelligence
• Natural Language Processing
• Computer Vision

Background:

• Social media and online memes necessitate advanced multimodal data analysis systems.
• Memes combine text and visuals, often conveying complex emotions, satire, or harmful content like sexism and bigotry.
• Existing AI struggles to classify nuanced and potentially offensive content within memes.

Purpose of the Study:

• To develop and evaluate a novel vision-language model for multitask meme categorization.
• To improve the automated analysis and classification of multimodal data in online forums.
• To address the challenge of identifying subtle and harmful content in memes.

Main Methods:

• Proposed a vision-language model named ViT-BERT CAMT (cross-attention multitask).
• Employed a linear self-attentive fusion mechanism to integrate Vision Transformer (ViT) image features and BERT text features.
• Evaluated the model on the SemEval 2020 Memotion and MIMIC datasets for sentiment, sarcasm, offensiveness, sexism, objectification, and prejudice detection.

Main Results:

• The ViT-BERT CAMT model achieved high accuracy on both tested datasets.
• The model demonstrated superior performance compared to existing baselines in multitask meme classification.
• Results confirm the efficacy of combined image-text modeling for nuanced meme interpretation.

Conclusions:

• Multimodal meme categorization is crucial for understanding online discourse and identifying harmful content.
• The ViT-BERT CAMT model offers a significant advancement in analyzing complex visual-textual data.
• This research contributes to better monitoring and comprehension of online conversations through improved AI.