Human-Guided Modality Informativeness for Affective States.

Area of Science:

• Affective computing
• Human-computer interaction
• Psychology

Background:

• Predicting affective states is crucial for understanding mental health, particularly depression.
• Multimodal approaches (vision, language, acoustic) are common in affect recognition.
• Understanding human perception of modality importance is key for developing interpretable AI.

Purpose of the Study:

• To test if all modalities are always necessary for predicting affective states.
• To investigate human perception of modality informativeness for affect recognition.
• To guide automatic affect recognition systems using human annotations for improved interpretability and performance.

Main Methods:

• Conducted a human study to assess modality informativeness for recognizing positive, aggressive, and dysphoric states.
• Developed and evaluated automatic affect recognition models guided by human annotations.
• Compared model interpretability and predictive performance with and without human guidance.

Main Results:

• Humans can reliably identify which modalities are most informative for affect recognition.
• Guided affect recognition models demonstrate improved interpretability, aligning with human judgments.
• Guided models show a slight increase in predictive performance compared to unguided models.

Conclusions:

• Not all modalities are required for accurate affect recognition; human perception can guide model development.
• Human-guided affect recognition systems offer enhanced interpretability without sacrificing performance.
• This approach has implications for developing more transparent and effective AI in mental health applications.