EndoChat: Grounded multimodal large language model for endoscopic surgery

Area of Science:

• Artificial Intelligence
• Medical Robotics
• Computer Vision

Background:

• Multimodal Large Language Models (MLLMs) show promise in computer-aided diagnosis and robotic-assisted surgery.
• A gap exists in specialized MLLMs for endoscopic surgical scene understanding.

Purpose of the Study:

• To introduce EndoChat, an MLLM designed for understanding endoscopic procedures.
• To develop a specialized dataset and novel mechanisms for improved surgical scene analysis.

Main Methods:

• Constructed the Surg-396K dataset using a novel pipeline for surgical information extraction and annotation.
• Implemented a multi-scale visual token interaction mechanism.
• Introduced a visual contrast-based reasoning mechanism for enhanced learning.

Main Results:

• Achieved state-of-the-art performance on five dialogue paradigms and seven surgical scene understanding tasks.
• Surgeons provided positive feedback on EndoChat's conversational case generation.
• Demonstrated potential for advancing training and automation in robotic-assisted surgery.

Conclusions:

• EndoChat represents a significant advancement in MLLMs for surgical applications.
• The developed dataset and model can accelerate progress in robotic-assisted surgery training and automation.
• Public availability of the dataset and model facilitates further research and development.