Dual modality prompt learning for visual question-grounded answering in robotic surgery

Area of Science:

• Robotic Surgery
• Computer Vision
• Artificial Intelligence

Background:

• Advancements in robotic surgery have spurred progress in visual question answering (VQA).
• Current VQA systems lack the ability to localize relevant image regions, limiting interpretability and exploration in surgical contexts.

Purpose of the Study:

• To propose a grounded VQA model for robotic surgery that can localize specific regions during answer prediction.
• To enhance multimodal information interactions for precise localization and accurate answer inference.

Main Methods:

• Developed a dual-modality prompt model inspired by prompt learning in language models.
• Introduced visual and textual complementary prompters for integrating visual and textual information.
• Employed a multiple iterative fusion strategy for comprehensive answer reasoning.

Main Results:

• The proposed grounded VQA model demonstrated superior performance compared to existing methods.
• The model effectively integrates visual and textual prompts for accurate localization and answer generation.
• Experiments were validated on the EndoVis-18 and EndoVis-17 datasets.

Conclusions:

• The grounded VQA model significantly improves localization and answer prediction in robotic surgery.
• The dual-modality prompt approach enhances multimodal interactions for VQA tasks.
• This model offers a more interpretable and exploratory VQA solution for surgical applications.