Unsupervised Modality-Transferable Video Highlight Detection With Representation Activation Sequence Learning.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Manual video labeling is time-consuming and limits supervised learning for new video categories.
• Many videos lack audio, hindering multimodal approaches for highlight detection.
• Efficient highlight detection is vital for video editing on internet platforms.

Purpose of the Study:

• To develop a novel unsupervised model for highlight detection using cross-modal perception.
• To address limitations of supervised methods and audio absence in highlight detection.
• To improve the efficiency of video editing by automating highlight identification.

Main Methods:

• A novel model with cross-modal perception for unsupervised highlight detection.
• Learning visual-audio semantics via self-reconstruction with image-audio pairs.
• Representation Activation Sequence Learning (RASL) module with k-point contrastive learning for significant activations.
• Symmetric Contrastive Learning (SCL) module for paired visual-audio representation learning.
• Auxiliary masked Feature Vector Sequence (FVS) reconstruction for representation enhancement.

Main Results:

• The proposed model achieves superior performance in unsupervised highlight detection compared to state-of-the-art methods.
• The model effectively generates representations with paired visual-audio semantics from visual modality alone during inference.
• RASL module successfully outputs highlight scores for video segments.

Conclusions:

• The developed unsupervised cross-modal highlight detection framework offers a significant advancement.
• The model overcomes challenges posed by manual labeling and audio absence.
• This approach enhances video editing efficiency by automating the identification of key moments.