Learning Social Spatio-Temporal Relation Graph in the Wild and a Video Benchmark

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Existing social relation recognition studies primarily use still images or movie clips, which do not reflect real-world complexities.
• Movie-based datasets often simplify the task to video classification, recognizing only one relation per scene, unlike dynamic, multi-relational real-world scenarios.

Purpose of the Study:

• To address the limitations of current methods by studying social relation recognition in open, real-world environments.
• To introduce the first video dataset, Social Relation In The Wild (SRIW), capturing diverse and numerous social interactions.
• To develop a robust framework capable of recognizing multiple social relations between individuals in complex scenes.

Main Methods:

• Proposed a Spatio-Temporal Relation Graph Convolutional Network (STRGCN) architecture to intuitively recognize social relations using visual features.
• Decoupled the recognition task into person-level and pair-level classification.
• Introduced a person behavior and character module for encoding static and dynamic features, forming a relation graph with meaningful edges for Graph Convolutional Network (GCN) and local GCN analysis.

Main Results:

• Achieved 83.1% mean Average Precision (mAP) in person-level relation classification.
• Attained 40.8% mAP in pair-level relation classification.
• Demonstrated the effectiveness and practicality of the proposed STRGCN framework in real-world scenarios.

Conclusions:

• The proposed STRGCN framework significantly advances social relation recognition in open environments.
• The SRIW dataset provides a valuable resource for future research in this domain.
• The study highlights the potential for practical applications of advanced AI in understanding complex human interactions.