Deep Modelling Strategies for Human Confidence Classification using Audio-visual Data.

Area of Science:

• Human-computer interaction
• Behavioral analysis
• Machine learning

Background:

• Human confidence is expressed through dynamic vocal and facial cues.
• These cues, while not always synchronized, influence each other and the overall expression.
• Analyzing time-varying expressions is crucial for understanding human behavior in contexts like interviews.

Purpose of the Study:

• To develop a deep fusion technique combining vocal and facial modalities for inferring human confidence.
• To analyze the temporal dynamics of confidence expressions in interview settings.
• To improve the classification performance of confident versus non-confident human behavior.

Main Methods:

• Collected interview data from 51 speakers.
• Applied a deep fusion technique to integrate speech and facial expression data.
• Utilized 5-fold cross-validation for performance analysis.

Main Results:

• Uni-modal models achieved an average Area Under the Curve (AUC) of 70.6% (speech) and 69.4% (facial expressions).
• The proposed deep fusion model significantly improved classification performance, achieving an average AUC of 76.8%.
• Demonstrated the effectiveness of fusing multi-modal temporal information for confidence inference.

Conclusions:

• Deep fusion of vocal and facial cues offers a superior method for inferring human confidence compared to uni-modal approaches.
• The model captures essential temporal dynamics for accurate behavior analysis.
• This approach has potential applications in various fields requiring reliable human behavior assessment.