Pair Then Relation: Pair-Net for Panoptic Scene Graph Generation.

Area of Science:

• Computer Vision
• Artificial Intelligence

Background:

• Panoptic Scene Graph (PSG) generation aims for comprehensive scene representations using pixel-level panoptic segmentation, unlike traditional Scene Graph Generation (SGG).
• Current PSG methods face challenges with pixel-level outputs and exploring all relationships (thing-thing, thing-stuff, stuff-stuff), leading to suboptimal performance.

Purpose of the Study:

• To design a novel and robust baseline for Panoptic Scene Graph generation.
• To address the performance limitations of existing PSG models and enhance downstream applications.

Main Methods:

• An in-depth analysis identified inter-object pair-wise recall as a critical bottleneck in current PSG models.
• A new framework, Pair-Net, was developed, incorporating a Pair Proposal Network (PPN) to learn and filter sparse pair-wise relationships.
• A lightweight Matrix Learner was designed within the PPN to directly learn pair-wise relationships for proposal generation.

Main Results:

• Pair-Net significantly enhances performance over a strong segmenter-based baseline.
• The proposed method achieved over 10% absolute performance gains compared to the state-of-the-art PSGFormer model.
• Extensive ablation studies validated the effectiveness of the Pair-Net framework and its components.

Conclusions:

• Inter-object pair-wise recall is a crucial, previously overlooked factor for effective Panoptic Scene Graph generation.
• Pair-Net provides a strong baseline and a significant improvement for the challenging PSG task.
• The developed framework offers a promising direction for advancing scene graph generation research.