Hierarchical Banzhaf Interaction for General Video-Language Representation Learning.

Area of Science:

• Artificial Intelligence
• Computer Vision
• Natural Language Processing

Background:

• Multimodal representation learning, particularly contrastive learning, is crucial in AI.
• Video-language representation learning currently relies on coarse-grained global semantic interactions.
• There's a need for fine-grained multimodal learning to enhance representation quality.

Purpose of the Study:

• To introduce a novel approach for fine-grained video-language representation learning.
• To model video-text interactions using multivariate cooperative game theory to handle uncertainty.
• To develop a method that captures diverse granularity, flexible combinations, and vague intensity in semantic interactions.

Main Methods:

• Modeling video-text as game players using multivariate cooperative game theory.
• Designing the Hierarchical Banzhaf Interaction (HBI) to simulate fine-grained correspondence between video clips and textual words from hierarchical perspectives.
• Reconstructing representations by fusing single-modal and cross-modal components to mitigate bias and preserve adaptive encoding.
• Extending the structure into a flexible encoder-decoder framework for adaptability to downstream tasks.

Main Results:

• The Hierarchical Banzhaf Interaction effectively simulates fine-grained semantic correspondence.
• Representation reconstruction mitigates bias, ensuring fine granularity and adaptive encoding.
• The encoder-decoder framework demonstrates flexibility across various tasks.
• Experiments on text-video retrieval, video-question answering, and video captioning benchmarks show superior performance.

Conclusions:

• The proposed method significantly enhances fine-grained video-language representation learning.
• The approach offers effective handling of uncertainty and diverse semantic interactions.
• The method demonstrates strong effectiveness and generalization capabilities on multiple benchmarks.