A Dual-Path Cross-Modal Network for Video-Music Retrieval

Area of Science:

• Artificial Intelligence
• Multimedia Retrieval
• Computer Vision

Background:

• Video and music integration is increasingly important but labor-intensive.
• Current video music retrieval methods often overlook crucial emotional nuances.
• Developing automated, emotionally aware music recommendation systems is a significant challenge.

Purpose of the Study:

• To propose an automated method for recommending background music for videos.
• To incorporate both content and emotional information for improved retrieval accuracy.
• To address the limitations of existing methods in capturing the emotional essence of music for videos.

Main Methods:

• A dual-path network architecture processing content and emotional information separately.
• Feature extraction using pre-trained networks and dimensionality reduction via encoder-decoder structures.
• Emotion path utilizes key frames for video and channel attention for music, enhanced by emotion distinguish loss.
• Fusion of content and emotional features through a shared MLP space with a polarity penalty factor.

Main Results:

• The proposed dual-path network effectively merges content and emotional information from video and music.
• Experimental results demonstrate a significant improvement in video music retrieval performance.
• Recall@1 metric increased by 3.94% compared to existing state-of-the-art methods.

Conclusions:

• The dual-path approach offers a superior method for video music retrieval by integrating multimodal information.
• The system successfully captures and utilizes emotional cues for more accurate music recommendations.
• This research advances the field of automated multimedia content analysis and recommendation.