Disentangling Inter- and Intra-Video Relations for Multi-Event Video-Text Retrieval and Grounding

Area of Science:

• Computer Vision
• Artificial Intelligence
• Information Retrieval

Background:

• Current video-text retrieval methods struggle with multi-event queries and fail to locate specific events within videos.
• Existing approaches are limited to single-text queries and do not provide precise event localization.
• There is a need for methods that can handle complex, multi-event textual descriptions and ground these events in video content.

Purpose of the Study:

• To propose a novel method for jointly addressing multi-event video-text retrieval and grounding.
• To enhance the precision of video-text alignment by considering inter- and intra-video event relationships.
• To accurately locate and ground multiple events within retrieved videos based on textual queries.

Main Methods:

• Developed a Relational Event-Centric Video-Text Retrieval module utilizing hierarchical event relationships for multi-level contrastive learning.
• Introduced Event Contrast-Driven Video Grounding to precisely locate multiple events by accounting for their positions on a temporal score map.
• Leveraged both inter-video and intra-video event relationships to improve retrieval and grounding performance.

Main Results:

• The proposed method significantly outperforms existing approaches on the ActivityNet Captions and Charades-STA benchmark datasets.
• Achieved superior performance in both multi-event video-text retrieval and precise event grounding.
• Demonstrated the effectiveness of the joint framework in handling complex textual queries and localizing multiple events.

Conclusions:

• The novel joint framework effectively addresses limitations in multi-event video-text retrieval and grounding.
• The method offers a significant advancement in searching and understanding video content based on complex textual descriptions.
• This research provides a foundation for future advancements in video analysis and retrieval applications.