Modality attention fusion model with hybrid multi-head self-attention for video understanding

Area of Science:

• Artificial Intelligence
• Computer Vision
• Natural Language Processing

Background:

• Video question answering (Video-QA) is a critical task for evaluating AI capabilities.
• Existing methods often struggle to effectively integrate multimodal information from video, subtitles, and questions.

Purpose of the Study:

• To propose a novel framework, Modality Attention Fusion with Hybrid Multi-head Self-attention (MAF-HMS), for enhancing video-QA performance.
• To effectively fuse attention and self-attention mechanisms across different modalities (video, subtitles, QA).

Main Methods:

• Utilized BERT for text feature extraction and Faster R-CNN for visual feature extraction.
• Developed a Modality Attention Fusion (MAF) framework to integrate features from video, subtitles, and QA.
• Employed Hybrid Multi-headed Self-attention (HMS) to determine the correct answer.

Main Results:

• The proposed MAF-HMS model significantly outperformed baseline methods on three scene datasets.
• Ablation studies confirmed the effectiveness of individual network components.
• Experimental results demonstrated advantages across different question types and required modalities.

Conclusions:

• The MAF-HMS framework offers a robust and effective approach to video question answering.
• The fusion of attention and self-attention mechanisms across modalities is crucial for improving AI's ability to understand and answer questions about videos.