SGT++: Improved Scene Graph-Guided Transformer for Surgical Report Generation

Area of Science:

• Medical Informatics
• Computer Vision
• Artificial Intelligence

Background:

• Automated surgical recording and report generation are vital for reducing surgeon workload and enhancing operational focus.
• Existing methods struggle to model instrument-tissue interactions and capture fine-grained differences in surgical images.

Purpose of the Study:

• To develop an improved scene graph-guided Transformer (SGT++) for more accurate surgical report generation.
• To address limitations in modeling interactive relationships and fine-grained visual details in surgical procedures.

Main Methods:

• Proposed an improved scene graph-guided Transformer (SGT++) to learn explicit and implicit interactions between surgical instruments and tissue.
• Developed a method for homogenizing heterogeneous scene graphs for graph learning.
• Introduced an attention-induced graph transformer for explicit relation-aware encoding and implicit relational attention using prototype memory.

Main Results:

• The SGT++ model effectively learns explicit and implicit relationships between surgical instruments and tissue.
• Fused relation-aware representations were obtained by coalescing explicit and implicit information.
• Achieved state-of-the-art results on two surgical datasets.

Conclusions:

• The SGT++ model demonstrates significant improvements in automated surgical report generation.
• Accurate modeling of instrument-tissue interactions and fine-grained visual details is crucial for enhanced performance.
• The proposed approach offers a promising solution for alleviating surgeons' workload and improving surgical documentation.