An image and text-based fake news detection with transfer learning

Area of Science:

• Artificial Intelligence
• Computer Science
• Information Science

Background:

• Fake news poses a significant threat to information reliability in the digital age.
• Current fake news detection methods often overlook visual content, which can be crucial for identifying misinformation.
• Limited labeled data presents a major challenge for training effective fake news detection models.

Purpose of the Study:

• To propose a multimodal classification approach for enhanced fake news detection.
• To address the challenge of data scarcity in fake news detection using efficient fine-tuning techniques.
• To evaluate the effectiveness of combining textual and visual information for improved accuracy.

Main Methods:

• Leveraged CLIP (Contrastive Language-Image Pre-training) for joint text-image feature extraction.
• Employed LoRA (Low-Rank Adaptation), a parameter-efficient fine-tuning method, to adapt CLIP for fake news detection.
• Utilized a simple one-layer multi-layer perceptron (MLP) for classification of multimodal features.

Main Results:

• Achieved 83% accuracy in classifying fake news using the multimodal approach with LoRA fine-tuning.
• Demonstrated the effectiveness of multimodal learning in improving fake news detection performance.
• Showcased the benefits of parameter-efficient fine-tuning techniques in low-resource settings.

Conclusions:

• Multimodal learning, integrating text and image data, significantly enhances fake news detection capabilities.
• Parameter-efficient fine-tuning techniques like LoRA are vital for developing robust fake news detectors with limited data.
• The proposed approach offers a promising solution for improving the reliability of online information.