Exploring the Essence of Relationships for Scene Graph Generation via Causal Features Enhancement Network

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Scene graph generation (SGG) is crucial for visual perception and reasoning, representing objects and their relationships.
• Current SGG methods often rely on language priors or statistical knowledge, which may not accurately reflect semantic interactions.
• A gap exists in capturing the true semantic essence of relationships, which should stem from object interactions.

Purpose of the Study:

• To propose a novel Causal Features Enhancement Network (CFEN) for more accurate scene graph generation.
• To mine essential semantic features between objects and relationships, moving beyond statistical dependencies.
• To enhance the understanding of object interactions for improved relationship recognition in SGG.

Main Methods:

• Decomposing object features into class-generic and object-specific components.
• Employing a causal graph framework to analyze existing SGG methods and measure the influence of object-specific features.
• Implementing a counterfactual training framework to compute differences between factual and counterfactual logits.
• Introducing a distribution matching loss using KL divergence to modulate relation predictions based on counterfactual outputs.

Main Results:

• The proposed CFEN effectively mines essential semantic features for relationship recognition.
• The counterfactual training framework successfully measures the influence of object-specific features.
• Experimental results on VG150 and VrR-VG datasets demonstrate the superiority of CFEN over state-of-the-art methods.

Conclusions:

• CFEN offers a significant advancement in scene graph generation by focusing on causal semantic features.
• The method provides a more accurate representation of object interactions compared to traditional SGG approaches.
• CFEN establishes a new benchmark for SGG performance on established datasets.