Tackling the Challenges in Scene Graph Generation With Local-to-Global Interactions

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Scene graph generation (SGG) faces challenges with ambiguity, asymmetry, and context.
• Existing SGG methods often overlook the directional nature of relationships.
• Leveraging higher-order contextual information is crucial for accurate scene graph prediction.

Purpose of the Study:

• To develop a novel SGG framework, LOGINs, that addresses key challenges in the task.
• To improve the understanding and generation of visually and semantically rich scene graphs.
• To introduce a new diagnostic task for evaluating relational direction awareness.

Main Methods:

• Designed Local-to-global interaction networks (LOGINs) for SGG.
• Incorporated direction awareness by constraining input order of subject and object.
• Utilized global interactions for context encoding and Attract and Repel loss for predicate embedding fine-tuning.
• Proposed Bidirectional Relationship Classification (BRC) task to quantify direction awareness.

Main Results:

• LOGINs framework demonstrates enhanced ability to distinguish relational direction compared to existing methods.
• Achieved state-of-the-art performance on the Visual Genome (VG) benchmark for SGG.
• LOGINs effectively leverages local and global interactions for improved scene graph prediction.

Conclusions:

• The LOGINs framework offers a significant advancement in scene graph generation.
• Addressing relational asymmetry and incorporating contextual information are key to improving SGG.
• The proposed BRC task provides a valuable metric for evaluating directional understanding in SGG models.