EmoShiftNet: a shift-aware multi-task learning framework with fusion strategies for emotion recognition in...

Area of Science:

• Artificial Intelligence
• Natural Language Processing
• Machine Learning

Background:

• Emotion Recognition in Conversations (ERC) is crucial for AI applications but struggles with detecting emotion shifts in multi-party conversations (MPCs).
• Existing models often overlook subtle, context-dependent emotion transitions, exacerbated by imbalanced datasets like MELD.
• Detecting these shifts is vital for nuanced conversational AI.

Purpose of the Study:

• To propose EmoShiftNet, a novel shift-aware multi-task learning (MTL) framework for improved ERC in MPCs.
• To jointly address emotion classification and emotion shift detection.
• To enhance the robustness of ERC models by explicitly modeling emotional transitions.

Main Methods:

• EmoShiftNet integrates multimodal features: BERT text embeddings, acoustic features (MFCCs, pitch, loudness), and temporal cues (pause duration, speaker overlap, utterance length).
• A shift-aware MTL framework performs joint emotion classification and shift detection.
• A composite loss function (focal loss, binary cross-entropy, triplet margin loss) incorporates emotion shift detection as an auxiliary task.

Main Results:

• EmoShiftNet outperformed traditional and graph-based ERC models on the MELD dataset, achieving higher F1-scores.
• The framework demonstrated improved recognition of minority emotions in imbalanced data conditions.
• Multimodal fusion and explicit shift supervision proved effective for enhancing ERC performance.

Conclusions:

• Modeling emotional transitions is critical for advancing ERC in multi-party conversations.
• EmoShiftNet's MTL approach with explicit shift detection enhances contextual awareness and robustness.
• This framework offers a more comprehensive solution for recognizing emotions in complex dialogues.