Multimodal machine learning for deception detection using behavioral and physiological data.

Area of Science:

• Cognitive Science
• Machine Learning
• Forensic Science

Background:

• Deception detection is vital for national security, judiciary, and legal systems.
• Traditional polygraphs face scientific and ethical challenges.
• Existing datasets for deception detection are limited, small, unimodal, and non-representative of diverse populations.

Purpose of the Study:

• To introduce CogniModal-D, a novel, real-world, multimodal dataset for deception detection tailored to the Indian population.
• To address the limitations of existing datasets in terms of size, modality, and population diversity.
• To evaluate the efficacy of multimodal AI approaches for automated deception detection.

Main Methods:

• Collected data across seven modalities: electroencephalography (EEG), electrocardiography (ECG), electrooculography (EOG), eye-gaze, galvanic skin response (GSR), audio, and video from over 100 Indian subjects.
• Utilized tasks involving social relationships and controlled mock crime interrogations.
• Developed and applied a multimodal AI-based score-level fusion technique to integrate diverse cues.

Main Results:

• Multimodal fusion significantly improved deception detection accuracy by up to 15% in mock crime and best friend scenarios compared to unimodal methods.
• Behavioral modalities (audio, video, gaze, GSR) demonstrated greater robustness than neurophysiological modalities (EEG, ECG, EOG).
• The CogniModal-D dataset provides a valuable resource for advancing deception detection research.

Conclusions:

• Multimodal features offer superior discriminatory power for deception detection.
• Integrating multiple data streams is crucial for developing robust and scalable deception detection systems.
• The findings support the development of advanced AI-driven lie detection technologies.