Fusion of Multi-Modal Features to Enhance Dense Video Caption

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Dense video captioning aims to generate descriptive text for video content.
• Existing methods often overlook crucial audio information, relying solely on visual features.
• This limitation hinders comprehensive video content analysis.

Purpose of the Study:

• To develop an advanced fusion model for dense video captioning.
• To integrate both visual and audio features for more accurate video understanding.
• To improve the performance of video captioning systems.

Main Methods:

• A fusion model employing the Transformer framework to integrate visual and audio features.
• Utilizing multi-head attention to manage varying sequence lengths.
• Implementing a Common Pool for feature alignment, redundancy filtering, and confidence scoring.
• Employing an LSTM (Long Short-Term Memory) network as a decoder for sentence generation.

Main Results:

• The proposed fusion model demonstrates competitive performance on the ActivityNet Captions dataset.
• Integration of audio features alongside visual features enhances captioning accuracy.
• The model effectively handles variations in sequence lengths and filters redundant information.

Conclusions:

• The fusion model offers a significant advancement in dense video captioning.
• Combining visual and audio data provides a more holistic approach to video analysis.
• The method shows promise for real-world applications requiring detailed video understanding.