Music video emotion classification using slow-fast audio-video network and unsupervised feature representation

Area of Science:

• Affective computing
• Multimodal machine learning
• Music information retrieval

Background:

• Emotion analysis is challenging due to subjectivity and the complex multimodal nature of music videos (lyrics, audio, visuals).
• Limited research and lack of standard datasets exist for music video emotion analysis.

Purpose of the Study:

• To propose an unsupervised method for music video emotion analysis.
• To create a labeled dataset for comparison.
• To evaluate multimodal architectures for audio-visual information fusion.

Main Methods:

• Developed an unsupervised method for music video emotion analysis using internet-sourced content.
• Employed a multimodal architecture with audio-video information exchange and boosting.
• Compared 2D/3D convolution networks and slow-fast networks with separable convolutions.
• Trained and evaluated supervised and unsupervised networks end-to-end.

Main Results:

• Unsupervised features and information sharing significantly improved classification scores.
• The best classifier achieved 77% accuracy, 0.77 F1-score, and 0.94 AUC.
• Demonstrated effective quantitative and qualitative interpretation of results on a large dataset.

Conclusions:

• The proposed unsupervised method offers a promising approach for music video emotion analysis.
• Multimodal fusion and information sharing enhance classification performance.
• The developed dataset and methods advance the field of affective computing for music videos.