Graph-based social relation inference with multi-level conditional attention.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Social relation inference from images requires advanced semantic understanding.
• Existing methods often focus on object detection, limiting nuanced relation analysis.
• Adaptive attention to scenes, objects, and interactions is crucial for accurate inference.

Purpose of the Study:

• To introduce a novel MUlti-level Conditional Attention (MUCA) mechanism for social relation inference.
• To develop an effective transformer-style network, the Graph-based Relation Inference Transformer (GRIT), for implementing MUCA.
• To outperform existing social relation inference methods using the proposed approach.

Main Methods:

• Proposed a MUlti-level Conditional Attention (MUCA) mechanism that considers scenes, objects, and human interactions per person pair.
• Developed the Graph-based Relation Inference Transformer (GRIT), a novel network architecture.
• GRIT comprises a Conditional Query Module (CQM) for generating relation queries and a Relation Attention Module (RAM) utilizing transformer-style multi-level attention.

Main Results:

• The proposed GRIT model significantly outperforms existing methods on two benchmark datasets.
• Achieved performance improvements of 7.8% on the PIPA dataset and 9.6% on the PISC dataset.
• Demonstrated the effectiveness of multi-level conditional attention for social relation inference.

Conclusions:

• GRIT is the first model to employ multi-level conditional attention for social relation inference.
• The proposed approach offers a more comprehensive understanding of social dynamics in images.
• GRIT provides a robust and accurate framework for social relation inference tasks.