Spatial-Temporal Knowledge-Embedded Transformer for Video Scene Graph Generation.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Video scene graph generation (VidSGG) involves identifying objects and their relationships within video frames.
• Understanding object interactions and temporal dynamics is crucial for accurate VidSGG.
• Spatial co-occurrence and temporal transition correlations offer valuable prior knowledge for VidSGG.

Purpose of the Study:

• To propose a novel Spatial-Temporal Knowledge-Embedded Transformer (STKET) for improved VidSGG.
• To effectively incorporate prior spatial-temporal knowledge into transformer-based models.
• To enhance the learning of representative relationship representations in videos.

Main Methods:

• Statistical learning of spatial co-occurrence and temporal transition correlations.
• Designing spatial and temporal knowledge-embedded layers utilizing multi-head cross-attention.
• Integrating visual representations with learned knowledge for embedded representations.
• Aggregating subject-object pair representations for final prediction.

Main Results:

• The proposed STKET model demonstrated superior performance compared to existing algorithms.
• Significant improvements in mR@50 were observed across various experimental settings (e.g., 8.1%, 4.7%, 2.1%).
• The method effectively leverages spatial-temporal priors for more accurate relationship inference.

Conclusions:

• STKET provides a powerful framework for video scene graph generation by embedding spatial-temporal knowledge.
• The integration of prior knowledge into the attention mechanism leads to more representative relationship predictions.
• The proposed approach represents a significant advancement in the field of VidSGG.