EAC-Agent: A deep learning framework for multimodal emotion-aware conversational agent with contextual response...

Area of Science:

• Artificial Intelligence
• Natural Language Processing
• Human-Computer Interaction

Background:

• Current conversational agents primarily use text, limiting their emotional understanding.
• There's a growing need for multimodal inputs (text, audio, video) in conversations.
• Existing agents struggle with generating emotionally accurate and contextually appropriate responses.

Purpose of the Study:

• To develop a novel multimodal conversational agent capable of understanding and responding to user emotions.
• To enhance emotional intelligence in chatbots by incorporating text, audio, and visual features.
• To address the limitations of single-modality inputs in current conversational AI.

Main Methods:

• Proposed the EAC-Agent, a sequence-to-sequence model utilizing transformers and pre-trained embeddings (e.g., GloVe).
• Implemented self and cross-modal attention mechanisms for text, audio, and visual features.
• Validated the model on two benchmark datasets: IEMOCAP and MELD.

Main Results:

• Achieved high accuracy in emotion recognition: 76.27% on IEMOCAP and 67.57% on MELD.
• Demonstrated improved emotion-aware response generation with low perplexity and high BLEU/ROUGE-L scores.
• EAC-Agent outperformed existing techniques in both emotion classification and response generation.

Conclusions:

• The EAC-Agent offers a superior multimodal approach for emotionally intelligent conversational AI.
• This model holds significant promise for applications requiring empathetic interactions, such as customer service and healthcare.
• The integration of multimodal features enhances the accuracy and appropriateness of chatbot responses.