An Audiovisual Correlation Matching Method Based on Fine-Grained Emotion and Feature Fusion

Area of Science:

• Artificial Intelligence
• Computer Vision
• Affective Computing

Background:

• Existing intelligent editing tools use cross-modal matching based on affective consistency or feature similarity.
• These methods struggle with complex audiovisual scenarios, leading to low accuracy and poor audience perception.

Purpose of the Study:

• To propose a hybrid matching model for artistic audiovisual works focusing on affective distribution similarity and integration.
• To enhance the accuracy and subjective experience of audiovisual matching in music and video editing.

Main Methods:

• Developed a hybrid model combining Canonical Correlation Analysis (CCA) and fine-grained affective similarity.
• Refined KCCA fusion features using matched and unmatched music-video pairs.
• Employed XGBoost for relevance prediction, incorporating affective semantic and factor distances.

Main Results:

• The proposed model achieves high prediction accuracy in audiovisual matching.
• Experimental results show a better subjective experience of audiovisual association.
• The model effectively balances feature parameters and affective semantic cognitions.

Conclusions:

• The novel affective matching model offers a new technical approach for music-video retrieval and editing.
• This research explores audiovisual affective association mechanisms from a sensory perspective.
• The findings contribute to improving intelligent tools for audiovisual content creation.