Synergistic Prompting Learning for Human-Object Interaction Detection.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Human-Computer Interaction

Background:

• Human-Object Interaction (HOI) detection is crucial for understanding real-world scenarios.
• Current HOI detectors use pre-trained Visual-Language Models (VLMs) with fixed textual prompts, limiting performance and generalization.
• Existing methods struggle with diverse HOIs in open-world contexts due to reliance on predetermined descriptive texts.

Purpose of the Study:

• To propose a novel VLM-based method for HOI detection that overcomes limitations of fixed textual prompts.
• To enhance HOI detection by jointly learning from visual and textual perspectives through synergistic prompting.
• To improve the performance and generalization capabilities of HOI detectors.

Main Methods:

• A hierarchical adaptation architecture for progressive visual and textual prompting.
• Visual prompting via gradual token migration from the VLM's image encoder.
• Textual prompting initialized with progressively leveled interaction descriptions.
• A text-supervising and image-tuning loop to synergize prompting learning.
• Interaction-aware knowledge merging for effective transfer of visual-textual knowledge.

Main Results:

• The proposed method outperforms state-of-the-art approaches on two benchmarks.
• Demonstrated superior performance in both supervised and zero-shot HOI detection settings.
• Achieved improved generalization capabilities compared to existing methods.

Conclusions:

• The synergistic visual-textual prompting approach significantly advances HOI detection.
• The method offers a more flexible and effective way to leverage VLMs for HOI tasks.
• This work provides a strong foundation for future research in open-world HOI understanding.