Scene graph-guided uncertainty decomposition improves confidence calibration in surgical visual question answering

Area of Science:

• Artificial Intelligence
• Medical Imaging
• Computer Vision

Background:

• Reliable confidence estimation is crucial for surgical visual question answering (VQA) to prevent overconfident errors in clinical decision support.
• Existing methods fail to capture hierarchical uncertainty from objects, tool-tissue interactions, and scene context.

Purpose of the Study:

• To propose a scene graph-guided framework for confidence-aware surgical VQA that decomposes and fuses hierarchical uncertainty.
• To improve probabilistic quality and enable selective answering through Dirichlet-based calibration.

Main Methods:

• Developed a framework decomposing uncertainty into object-level, relation-level, and scene-level components.
• Incorporated adaptive fusion of decomposed uncertainties.
• Utilized Dirichlet-based calibration for enhanced probabilistic quality.

Main Results:

• Achieved 63.58% overall accuracy and improved calibration performance (Brier Score, NLL) on the SSG-QA benchmark.
• Demonstrated significant gains for relation-type questions (+0.92%).
• Uncertainty decomposition boosted accuracy (+0.70%), while Dirichlet calibration improved probabilistic quality.

Conclusions:

• Explicitly modeling semantic-level uncertainty enhances reliability and interpretability in surgical VQA.
• The proposed framework offers fine-grained confidence information for safer AI-assisted surgical decision support.
• The SG-UD framework provides granular uncertainty interpretability and competitive calibration performance.