Towards zero-shot human-object interaction detection via vision-language integration.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Human-object interaction (HOI) detection is crucial for understanding complex scenes in images.
• Current supervised methods struggle with generalization to unseen object categories due to reliance on extensive annotations.
• Visual-language models (VLMs) show promise for zero-shot learning, offering a potential solution for HOI detection limitations.

Purpose of the Study:

• To develop a novel framework, Knowledge Integration to HOI (KI2HOI), for improved zero-shot HOI detection.
• To leverage the zero-shot capabilities of VLMs to enhance HOI detection performance.
• To address the generalization limitations of current HOI detection methods.

Main Methods:

• Proposed a novel framework, KI2HOI, integrating VLM knowledge for zero-shot HOI detection.
• Introduced a ho-pair encoder to provide contextual and interaction-specific semantic representations.
• Implemented two fusion strategies: visual-level fusion for global context and language-level fusion to enhance VLM capabilities.

Main Results:

• KI2HOI demonstrated superior performance compared to existing methods on HICO-DET and V-COCO datasets.
• The model achieved significant improvements in both zero-shot and full-supervised HOI detection settings.
• The proposed fusion strategies effectively facilitated prior knowledge transfer from VLMs.

Conclusions:

• The KI2HOI framework effectively integrates VLM knowledge to advance zero-shot HOI detection.
• The method shows strong generalization capabilities, outperforming previous approaches.
• This work offers a promising direction for more robust and adaptable HOI detection systems.