Multi-modal emotion recognition in conversation based on prompt learning with text-audio fusion features

Area of Science:

• Artificial Intelligence
• Natural Language Processing
• Human-Computer Interaction

Background:

• Interactive machine applications are increasing, driving demand for advanced Emotion Recognition in Conversations (ERC) technology.
• Current ERC methods struggle with language barriers and limited non-English data, hindering cross-lingual applications.
• Existing approaches often rely on single modalities, limiting comprehensive emotion understanding.

Purpose of the Study:

• To develop an innovative multimodal emotion recognition in conversations (MERC-PLTAF) method.
• To address the limitations of single modality and language barriers in current ERC technologies.
• To enhance cross-lingual emotion recognition capabilities.

Main Methods:

• Proposed the MERC-PLTAF method focusing on multimodal emotion recognition.
• Employed refined feature extraction techniques.
• Utilized a sophisticated cross-fusion strategy for integrating multimodal information.

Main Results:

• Demonstrated significant improvements in emotion recognition accuracy across multiple English and Chinese datasets.
• Achieved exceptional performance on the Chinese M3ED dataset, showcasing cross-lingual effectiveness.
• Validated the MERC-PLTAF method's superiority over existing approaches.

Conclusions:

• The MERC-PLTAF method offers a new pathway for effective cross-lingual emotion recognition.
• This research advances ERC technology and provides a framework for more intelligent human-computer interaction.
• The findings support the development of more human-centric interactive experiences through improved emotion understanding.