Multimodal transformer and directed relational graph learning for emotion recognition in conversations with potential...

Multimodal emotion recognition in conversations aims to identify the emotion expressed in each utterance by jointly analyzing textual, acoustic, visual, and contextual information. However, existing methods still face challenges in coordinating intra-modal dependency modeling, cross-modal interaction, and graph-based conversational reasoning. To address these challenges, this paper proposes MTG-ERC, a multimodal conversational emotion recognition framework that integrates transformer-based attention mechanisms with directed relational graph learning. First, textual, acoustic, and visual features are projected into a unified representation space. Intra-modal self-attention and cross-modal attention are then employed to capture modality-specific contextual dependencies and complementary interactions among modalities. Subsequently, a directed multi-relational graph module combines relational graph convolution and Graph Transformer operations to model temporal context, speaker-related interactions, and multimodal dependencies among utterances. Finally, global attention-based representations and local graph-based contextual representations are integrated for utterance-level emotion classification. Experiments on the IEMOCAP and MELD datasets show that MTG-ERC achieves competitive performance compared with the evaluated conversational emotion recognition baselines. Ablation results further indicate that intra-modal attention, cross-modal interaction, and directed relational modeling make complementary contributions to the final performance. The proposed framework also has potential applications in college student mental health research, where its multimodal affective modeling capability could support the analysis of emotional patterns in conversational data.