Explainable human-centered traits from head motion and facial expression dynamics.

Area of Science:

• Computer Science
• Psychology
• Human-Computer Interaction

Background:

• Predicting human-centered traits from behavior is crucial for various applications.
• Explainability in AI predictions enhances trust and understanding.
• Multimodal behavioral cues offer a rich source of information for trait prediction.

Purpose of the Study:

• To evaluate the efficacy of multimodal behavioral cues for explainable prediction of personality and interview-specific traits.
• To investigate different fusion strategies for combining unimodal and multimodal data.
• To assess the performance of Long Short-Term Memory (LSTM) architectures on trait prediction tasks.

Main Methods:

• Utilized kinemes (head-motion units), action units (facial movements), and speech features.
• Employed decision-level, feature-level, and attention-based fusion strategies.
• Applied various LSTM architectures for classification and regression on the MIT Interview and FICS datasets.

Main Results:

• Multimodal approaches significantly outperformed unimodal ones, achieving high prediction accuracy (e.g., 0.98 PCC for Excited-Friendly traits).
• Both unimodal and multimodal methods provided efficient trait predictions and plausible explanations.
• Effective trait prediction was achieved even with two-second behavioral snippets (thin-slice approach).

Conclusions:

• Multimodal behavioral cues are effective for explainable personality and interview trait prediction.
• Attention-based fusion effectively quantifies the importance of different modalities.
• The thin-slice approach demonstrates the potential for rapid behavioral analysis.