Bridging Visual and Textual Semantics: Towards Consistency for Unbiased Scene Graph Generation

Area of Science:

• Computer Vision
• Artificial Intelligence
• Cognitive Science

Background:

• Scene Graph Generation (SGG) aims to identify visual relationships within images.
• Existing SGG methods suffer from long-tailed bias, limiting their practical application.
• Current approaches often oversimplify SGG as a classification task, hindering fine-grained detail capture and increasing ambiguity.

Purpose of the Study:

• To propose a novel Visual-Textual Semantics Consistency Network (VTSCN) for Scene Graph Generation.
• To address and significantly alleviate the long-tailed bias in SGG.
• To model SGG as a reasoning process inspired by dual-process cognitive psychology.

Main Methods:

• Introduced a Hybrid Union Representation (HUR) module simulating the rapid, autonomous Type 1 cognitive process for spatial awareness and working memory.
• Developed a Global Textual Semantics Modeling (GTS) module for higher-order reasoning (Type 2 process) by modeling textual contexts of object pairs.
• Integrated a Heterogeneous Semantics Consistency (HSC) module to balance Type 1 and Type 2 processes, mimicking associative cognition.

Main Results:

• The VTSCN demonstrated superior performance compared to state-of-the-art methods on the Visual Genome, GQA, and PSG datasets.
• Ablation studies confirmed the effectiveness of the proposed VTSCN architecture and its components.
• The VTSCN successfully models SGG as a reasoning task, mitigating long-tailed bias.

Conclusions:

• The VTSCN offers a novel framework for SGG model design by incorporating human cognitive processes.
• This approach effectively reduces long-tailed bias, making SGG more practical.
• The findings suggest that cognitive psychology principles can inspire more robust and accurate computer vision models.