SGSG: Stroke-Guided Scene Graph Generation.

Area of Science:

• Computer Vision
• Human-Computer Interaction
• Robotics

Background:

• 3D scene graph generation is crucial for spatial computing in Extended Reality (XR).
• Current methods struggle with noisy point cloud data and lack interactive user guidance for improved accuracy.
• Key challenges include controllable interaction, integrating guidance into inference, and generalizing corrections.

Purpose of the Study:

• To develop an interactive method for refining 3D semantic scene graphs in real-time.
• To address limitations in accuracy caused by noisy 3D data and the absence of user input.
• To establish a new benchmark for interactive 3D scene graph generation in XR.

Main Methods:

• Proposed Stroke-Guided Scene Graph generation (SGSG) method.
• Introduced three stroke types and a specialized SGstrokes dataset.
• Integrated stroke guidance for spatio-temporal feature learning and reasoning correction.
• Utilized intervention losses with consistency-repulsive and geometry-sensitive constraints.

Main Results:

• SGSG demonstrated superior overall accuracy and precision compared to 3DSSG and SGFN.
• Outperformed JointSSG in predicate-level metrics.
• Significantly reduced user task load across all tested control conditions.
• Established SGSG as a new benchmark for interactive 3D scene graph generation.

Conclusions:

• SGSG effectively enables interactive refinement of 3D semantic relationships.
• The method improves accuracy and generalization in challenging 3D data conditions.
• SGSG offers a promising approach for enhanced semantic understanding in XR applications.