Multi-Stage Image-Language Cross-Generative Fusion Network for Video-Based Referring Expression Comprehension.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Natural Language Processing

Background:

• Video-based referring expression comprehension is challenging due to reliance on tracking templates or limited frame analysis.
• Existing methods struggle with feature fusion across multiple frames and integrating language cues effectively.

Purpose of the Study:

• To propose a novel network, MILCGF-Net, for improved video-based referring expression comprehension.
• To enhance the fusion of multi-frame features and cross-modal information for accurate object localization.

Main Methods:

• Developed a Multi-Stage Image-Language Cross-Generative Fusion Network (MILCGF-Net) based on one-stage object detection.
• Implemented a Frame Dense Feature Aggregation module for adjacent time sequence feature learning.
• Introduced an Image-Language Cross-Generative Fusion module with a consistency loss for refined cross-modal feature generation.

Main Results:

• The proposed MILCGF-Net demonstrates effectiveness in video-based referring expression comprehension.
• Comprehensive experiments on three public datasets validate the model's performance.

Conclusions:

• MILCGF-Net addresses limitations in existing methods by improving feature fusion and cross-modal understanding.
• The novel approach shows significant potential for accurate object localization in videos based on linguistic descriptions.