S²Former-OR: Single-Stage Bi-Modal Transformer for Scene Graph Generation in OR

Area of Science:

• Computer Vision
• Artificial Intelligence
• Medical Informatics

Background:

• Scene graph generation (SGG) is vital for operating room (OR) cognitive intelligence.
• Existing SGG methods often use multi-stage pipelines, limiting flexibility and effectiveness.
• These pipelines rely on intermediate pose estimation and object detection, compromising end-to-end learning.

Purpose of the Study:

• To introduce a novel single-stage bi-modal transformer framework, S2Former-OR, for surgical scene graph generation.
• To leverage multi-view 2D scenes and 3D point clouds for SGG in an end-to-end manner.
• To enhance the flexibility and effectiveness of multimodal representations in the OR.

Main Methods:

• Developed a single-stage bi-modal transformer framework (S2Former-OR).
• Implemented a View-Sync Transfusion scheme for multi-view visual information interaction.
• Designed a Geometry-Visual Cohesion operation to integrate 2D semantic and 3D point cloud features.
• Proposed a relation-sensitive transformer decoder for direct entity-pair relation prediction.

Main Results:

• S2Former-OR demonstrated superior SGG performance on the 4D-OR benchmark.
• Achieved a 3 percentage point increase in Precision compared to existing OR-SGG methods.
• Reduced model parameters by 24.2 million, indicating lower computational cost.
• Consistently outperformed generic single-stage SGG methods in comprehensive evaluations.

Conclusions:

• S2Former-OR offers an effective and efficient end-to-end solution for surgical scene graph generation.
• The proposed framework enhances multimodal representation learning for OR applications.
• This approach advances cognitive intelligence in the operating room through improved SGG.