End-to-End Open-Vocabulary Video Visual Relationship Detection Using Multi-Modal Prompting

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Current open-vocabulary video visual relationship detection methods rely on pre-trained trajectory detectors, limiting generalization to novel object categories.
• This dependence leads to performance degradation when encountering unseen objects and relationships.

Purpose of the Study:

• To develop an end-to-end open-vocabulary framework unifying object trajectory detection and relationship classification.
• To improve the generalization ability of video visual relationship detection systems to novel object categories and relationships.

Main Methods:

• Proposing a relationship-aware open-vocabulary trajectory detector using a query-based Transformer decoder with distilled CLIP visual encoder.
• Integrating a relationship query and auxiliary loss to explicitly perceive object relationships during trajectory detection.
• Developing an open-vocabulary relationship classifier with a multi-modal prompting method (spatio-temporal visual and vision-guided language prompting) to adapt CLIP.

Main Results:

• The proposed framework demonstrates effectiveness on VidVRD and VidOR datasets.
• The approach shows strong generalization ability in a challenging cross-dataset scenario.

Conclusions:

• The unified framework successfully addresses limitations of existing methods in open-vocabulary video visual relationship detection.
• The developed relationship-aware detector and classifier significantly enhance the detection of novel relationships and improve generalization.