Deep-Learning-Based Multimodal Emotion Classification for Music Videos.

Area of Science:

• Computer Science
• Artificial Intelligence
• Multimedia Analysis

Background:

• Music videos contain rich audio and visual information.
• Analyzing emotions in music videos requires processing multiple data types.
• Current affective computing methods may not fully leverage multimodal data.

Purpose of the Study:

• To develop an efficient multimodal affective computing system for music videos.
• To analyze emotions by integrating music, video, and facial expression cues.
• To reduce the computational cost of affective computing models.

Main Methods:

• Developed a multimodal system using music, video, and facial expression data.
• Applied audio-video information exchange and boosting for regularization.
• Utilized separable convolution for reducing computational expenses.
• Implemented information-sharing methods within multimodal representations.

Main Results:

• Multimodal representations effectively capture acoustic and visual emotional cues.
• Computational costs were significantly reduced via factorized convolutions.
• Information-sharing boosted individual data flow and overall performance.
• The best classifier achieved 74% accuracy, 0.73 F1-score, and 0.926 AUC.

Conclusions:

• Multimodal affective computing is crucial for comprehensive emotional analysis in music videos.
• The proposed system demonstrates efficiency in both performance and computational cost.
• Information-sharing strategies enhance the effectiveness of multimodal emotion recognition.