Detecting deception using machine learning with facial expressions and pulse rate.

Area of Science:

• Computer Science
• Psychology
• Human-Computer Interaction

Background:

• Remote interviews are prevalent due to the COVID-19 pandemic, increasing the need for reliable deception detection.
• Current methods for detecting deception in interviews rely heavily on interviewer experience and are not automated.
• Interviewees may struggle to convey truthfulness or may attempt deception during remote interactions.

Purpose of the Study:

• To develop a machine learning approach for automated deception detection in remote interviews.
• To integrate facial expression analysis with pulse rate data for enhanced detection accuracy.
• To create a realistic dataset for training and evaluating deception detection models.

Main Methods:

• A machine learning model was developed to correlate facial expression features with pulse rate data.
• A novel dataset was created using natural, improvised responses from subjects recorded via webcam and smartwatch.
• The model was evaluated using 10-fold cross-validation with a random forests classifier.

Main Results:

• The proposed machine learning approach demonstrated high accuracy, with F1 scores ranging from 0.75 to 0.88.
• Feature importance analysis revealed subject-specific indicators of deception.
• The model effectively combined facial and physiological data for deception detection.

Conclusions:

• The study successfully developed and validated a machine learning system for detecting deception in remote interviews.
• The findings highlight the potential of integrating multimodal data (facial expressions, pulse rate) for automated lie detection.
• Individualized feature analysis offers insights into the unique physiological and behavioral markers of deception across different subjects.