Towards Generalizable Deepfake Detection: An Inconsistency-Aware Vision-Language Model Tuning Approach

Area of Science:

• Artificial Intelligence
• Computer Vision
• Digital Forensics

Background:

• Advanced AI models generate deepfakes, challenging digital media authenticity.
• Large Vision-Language Models (VLMs) show promise for image forensics due to multimodal representations.
• Existing VLM methods often freeze visual encoders, potentially missing crucial low-level artifacts for deepfake detection.

Purpose of the Study:

• To propose IncoTune, an inconsistency-aware tuning framework for VLM-based deepfake detection.
• To enhance sensitivity to subtle inconsistencies indicative of deepfakes.
• To improve cross-dataset detection performance with efficient parameter tuning.

Main Methods:

• IncoTune framework with trainable vision-side adaptation for the visual encoder.
• Directional Low-Rank Adaptation (DoRA) applied to linear projection layers in both visual and language models.
• Empirical evaluation using datasets like FaceForensics++, DFD, Celeb-DF, DFDC, and DFDCP.

Main Results:

• IncoTune improves cross-dataset frame-level deepfake detection performance compared to frozen-vision baselines.
• DoRA combined with vision-side adaptation yields performance gains.
• Achieves competitive results with only 27.0M trainable adapter parameters.
• Demonstrates robustness under common image degradations.

Conclusions:

• IncoTune offers an effective and efficient approach to deepfake detection using VLMs.
• Trainable vision-side adaptation and DoRA are key to enhancing detection sensitivity.
• The framework provides a robust solution for authenticating digital media.