A Robust Approach to Multimodal Deepfake Detection.

Area of Science:

• Computer Science
• Artificial Intelligence
• Digital Forensics

Background:

• Deep learning enables realistic synthetic media (deepfakes), posing societal threats.
• Current deepfake detection methods struggle with multimodal consistency and temporal accuracy.
• Distinguishing authentic from fake media is increasingly vital.

Purpose of the Study:

• To propose a novel approach for detecting deepfake video sequences using data multimodality.
• To leverage inconsistencies between and within audio-visual data for enhanced detection.
• To develop a robust deepfake detector trained on disjoint monomodal datasets.

Main Methods:

• Extraction of time-aware audio-visual features from video sequences.
• Analysis using time-aware neural networks to identify inter-modal inconsistencies.
• Training on separate visual-only and audio-only deepfake datasets, testing on multimodal deepfakes.

Main Results:

• A multimodal approach significantly outperforms monomodal detection.
• The proposed method demonstrates robustness on unseen multimodal deepfakes.
• Investigated and compared different data fusion techniques for optimal performance.

Conclusions:

• Leveraging audio-visual inconsistencies is effective for deepfake detection.
• The proposed training strategy overcomes the limitation of scarce multimodal deepfake datasets.
• This method offers a robust solution for identifying sophisticated deepfake videos.