Toward a Unified Transformer-Based Framework for Scene Graph Generation and Human-Object Interaction Detection.

Area of Science:

• Computer Vision
• Artificial Intelligence

Background:

• Scene graph generation (SGG) and human-object interaction (HOI) detection are distinct visual recognition tasks.
• Current methods often develop task-specific models, limiting potential synergistic improvements.

Purpose of the Study:

• To investigate the intrinsic relationship between SGG and HOI detection.
• To develop a unified model that leverages scene graphs for enhanced HOI inference.

Main Methods:

• Introduced SG2HOI+, a one-step unified model based on the Transformer architecture.
• Employed two interactive hierarchical Transformers: one for relation triple generation and another for HOI prediction.
• Utilized visual features and generated relation triples to infer human-object interactions.

Main Results:

• SG2HOI+ demonstrated compelling performance compared to one-stage SGG models on benchmark datasets (Visual Genome, V-COCO, HICO-DET).
• The model achieved competitive results against state-of-the-art HOI detection methods.
• Joint end-to-end training for both SGG and HOI tasks yielded substantial improvements over individualized training.

Conclusions:

• Unifying SGG and HOI detection within a single framework enhances performance for both tasks.
• Scene graphs provide crucial relational cues that significantly improve human-object interaction inference.
• The proposed Transformer-based SG2HOI+ model offers an effective approach for integrated visual relationship and interaction understanding.