CLAAF: Multimodal fake information detection based on contrastive learning and adaptive Agg-modality fusion

Area of Science:

• Computer Science
• Artificial Intelligence
• Information Science

Background:

• Social media disinformation poses challenges for content authenticity verification, particularly in multimodal contexts.
• Simple fusion methods struggle with differing feature representations, leading to noise and reduced detection accuracy.

Purpose of the Study:

• To propose a novel model, Contrastive Learning and Adaptive Agg-modality Fusion (CLAAF), for improved multimodal fake information detection.
• To address the limitations of simple modality fusion in handling noise and enhancing accuracy.

Main Methods:

• Developed a contrastive learning strategy to align text and image modalities, preserving key features and reducing noise.
• Introduced an adaptive agg-modality fusion module for deep inter-modal interaction and integration.
• Constructed a comprehensive multimodal dataset from authoritative news and fact-checking platforms.

Main Results:

• The CLAAF model demonstrated a 3.45% improvement in accuracy over existing baseline models.
• Achieved enhanced precision and robustness in detecting multimodal fake information.
• The developed dataset provides a robust foundation for model training and validation.

Conclusions:

• The CLAAF model effectively mitigates noise and improves the accuracy of multimodal fake information detection.
• Adaptive fusion and contrastive learning are key to enhancing the model's performance.
• The findings contribute to more reliable content authenticity verification in the digital age.