Forensics Adapter: Unleashing CLIP for Generalizable Face Forgery Detection

Area of Science:

• Computer Vision
• Digital Forensics
• Machine Learning

Background:

• Adapting versatile models like CLIP for specialized tasks like face forgery detection is challenging due to entangled knowledge.
• Existing methods often use CLIP as a feature extractor without task-specific adaptation, limiting face forgery detection effectiveness.

Purpose of the Study:

• To develop an adapter network, Forensics Adapter, for transforming CLIP into an effective and generalizable face forgery detector.
• To introduce a method that learns face forgery traces and enhances CLIP's visual tokens through cross-modal interaction.

Main Methods:

• Developed Forensics Adapter, an adapter network designed to learn face forgery traces (e.g., blending boundaries) using task-specific objectives.
• Implemented a dedicated interaction strategy to enhance CLIP's visual tokens by communicating knowledge between CLIP and the adapter.
• Proposed Forensics Adapter++, incorporating textual modality via forgery-aware prompt learning.

Main Results:

• Forensics Adapter achieves superior performance across six standard datasets with only 5.7M trainable parameters.
• The Forensics Adapter++ extension, utilizing textual modality, provides an additional 1.3% performance boost.
• The proposed methods demonstrate strong generalizability due to the adapter's integration alongside CLIP.

Conclusions:

• Forensics Adapter offers an effective and generalizable solution for face forgery detection by adapting CLIP.
• The methods provide a strong baseline for future CLIP-based face forgery detection research.
• The code is publicly available to facilitate further research and development in digital forensics.