Toward Multimodal Modeling of Emotional Expressiveness

Area of Science:

• Psychology
• Computer Science
• Human-Computer Interaction

Background:

• Emotional expressiveness significantly impacts behavioral health and social interactions.
• Automatic prediction of emotional expressiveness can advance various scientific and industrial fields.

Purpose of the Study:

• To assess the predictability of emotional expressiveness from visual, linguistic, and multimodal behavioral signals.
• To determine the relative importance of each behavioral modality.
• To identify specific behavioral signals reliably associated with emotional expressiveness.

Main Methods:

• Utilized an existing video database augmented with reliable transcripts and human ratings of perceived emotional expressiveness.
• Trained, validated, and tested predictive models using visual, linguistic, and multimodal data.
• Employed interpretable models to identify key predictive signals.

Main Results:

• The best predictive model achieved promising performance (RMSE = 0.65, R² = 0.45, r = 0.74).
• Multimodal models generally yielded the best results.
• Linguistic modality models outperformed visual modality models.
• Specific signals like facial action unit intensity, word count, and social process words were reliable predictors.

Conclusions:

• Emotional expressiveness can be reliably predicted from behavioral signals.
• Multimodal and linguistic approaches offer superior predictive power.
• Interpretable models highlight specific visual and linguistic cues crucial for predicting emotional expressiveness.