Human Conversation Analysis Using Attentive Multimodal Networks with Hierarchical Encoder-Decoder.

Area of Science:

• Artificial Intelligence
• Human-Computer Interaction
• Natural Language Processing

Background:

• Human conversation analysis is complex due to multimodal communication (words, intonation, body language).
• Existing models often struggle to integrate diverse sensory inputs effectively.

Purpose of the Study:

• To develop a hierarchical encoder-decoder model with attention for comprehensive conversation analysis.
• To introduce a novel fusion strategy with modality attention for integrating multiple sensory inputs.
• To enhance the accuracy and interpretability of conversation analysis systems.

Main Methods:

• A hierarchical encoder-decoder structure was employed to learn word-level and conversation-level features.
• A novel fusion strategy with modality attention was developed to integrate video, audio, and text data.
• The system was evaluated on emotion recognition, sentiment analysis, and speaker trait analysis datasets.

Main Results:

• The proposed system outperformed state-of-the-art approaches in classification and regression tasks across three datasets.
• Superior generalization performance was observed on two common datasets.
• Comparable performance was achieved for predicting co-existing labels compared to multiple individual models.

Conclusions:

• The hierarchical model with attention effectively analyzes multimodal human conversations.
• The modality attention mechanism improves feature selection and model interpretability.
• This approach offers a robust and interpretable solution for complex conversation analysis.