Towards few-shot deepfake detection with an enhanced CLIP model

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Deepfakes pose significant threats to privacy, public trust, and democratic integrity.
• Current deepfake detection methods struggle with generalization to unseen forgery techniques due to reliance on extensive supervised learning data.
• Few-shot learning approaches show potential but have limited success in deepfake detection compared to conventional image classification.

Purpose of the Study:

• To develop a robust deepfake detection method effective in practical settings, particularly under few-shot conditions.
• To address the limitations of existing methods in generalizing across diverse and unrelated deepfake generation techniques.
• To enhance few-shot learning performance for fine-grained visual tasks like deepfake detection.

Main Methods:

• Proposing Instance-level Few-shot Prompt Learning (IFPL) within the CLIP framework.
• Enhancing the text branch with multi-scale adaptive context and instance-level facial embeddings.
• Introducing learnable visual perturbation blocks in the visual branch and a nonparametric prototype-based instance cache module for external guidance.

Main Results:

• IFPL demonstrates superior performance compared to state-of-the-art methods in few-shot deepfake detection scenarios.
• The proposed methodology effectively guides the encoder towards forgery-specific artifacts.
• The instance cache module provides crucial external guidance for robust decision-making.

Conclusions:

• IFPL establishes a novel and effective solution for deepfake detection, especially when training data is limited.
• The approach significantly improves generalization capabilities across different deepfake generation techniques.
• This work advances the field of few-shot learning for complex, fine-grained visual recognition tasks.