Advancing surgical VQA with scene graph knowledge.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Medical Informatics
• Surgical Data Science

Background:

• Modern operating rooms require advanced intra-operative support systems.
• Surgical data science is expanding beyond video analysis to integrate natural language processing.
• Current surgical visual question answering (VQA) systems face challenges with dataset bias and limited scene-aware reasoning.

Purpose of the Study:

• To advance surgical VQA by incorporating scene graph knowledge.
• To address question-condition bias in surgical VQA datasets.
• To develop a surgical VQA model with enhanced scene-aware reasoning capabilities.

Main Methods:

• A novel surgical scene graph-based dataset (SSG-VQA) was created using segmentation and detection models.
• Surgical scene graphs were constructed with spatial and action information of instruments and anatomies.
• A new VQA model, SSG-VQA-Net, was proposed, featuring a Scene-embedded Interaction Module for integrating geometric scene knowledge via cross-attention.

Main Results:

• The SSG-VQA dataset is more complex, diverse, geometrically grounded, unbiased, and action-oriented than existing datasets.
• SSG-VQA-Net demonstrated superior performance across various question types and complexities compared to existing methods.
• A lack of scene knowledge was identified as the primary limitation in current surgical VQA systems for complex queries.

Conclusions:

• Incorporating geometric scene features significantly improves surgical VQA model performance.
• The bottleneck in current surgical VQA models lies in learning encoded representations rather than sequence decoding.
• The SSG-VQA dataset serves as a benchmark for evaluating model scene understanding and reasoning capabilities.