In Defense of Clip-Based Video Relation Detection

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Video Visual Relation Detection (VidVRD) identifies visual relationship triplets in videos.
• Current methods use bottom-up (clip-based) or top-down (video-based) paradigms.
• Effective spatial and temporal context modeling is crucial for VidVRD performance.

Purpose of the Study:

• To revisit and improve the clip-based paradigm for VidVRD.
• To propose a novel Hierarchical Context Model (HCM) for enhanced context modeling.
• To demonstrate the superiority of clip-based methods with advanced context.

Main Methods:

• Developed a Hierarchical Context Model (HCM) focusing on clips.
• Enriched object-based spatial context and relation-based temporal context within clips.
• Utilized clip tubelets instead of video tubelets for relation classification.

Main Results:

• The proposed HCM achieved state-of-the-art performance on two VidVRD benchmarks.
• Clip-based methods with HCM outperformed most video-based approaches.
• HCM demonstrated advantages over video tubelets, avoiding long-term tracking issues.

Conclusions:

• Advanced spatial and temporal context modeling within the clip-based paradigm is highly effective for VidVRD.
• The clip-based approach with HCM offers flexibility and overcomes limitations of video tubelets.
• This work re-establishes the potential of clip-based methods in VidVRD.