Simultaneous Multi-Structure Segmentation and 3D Nonrigid Pose Estimation in Image-Guided Robotic Surgery

Area of Science:

• Medical Imaging
• Robotic Surgery
• Computer-Aided Surgery

Background:

• Accurate segmentation of endoscopic video is crucial for image-guided robotic surgery, providing surgeons with contextual information for real-time decision-making.
• Challenges in segmentation include noise, poor contrast, occlusions, and limited visibility, hindering perception of the surgical scene.

Purpose of the Study:

• To develop a multi-modal segmentation approach for endoscopic video in image-guided robotic surgery.
• To improve surgeons' perception and decision-making by providing enhanced contextual information during procedures.

Main Methods:

• A multi-modal approach jointly analyzing preoperative 3D computed tomography (CT) scans and intraoperative stereo-endoscopic video data.
• Estimating and tracking the pose of preoperative models, accounting for non-rigid deformations, to segment poorly visible structures in endoscopic videos.
• Embedding camera calibration parameters into the optimization process to correct for potential inaccuracies during surgery.

Main Results:

• Demonstrated high accuracy and robustness in segmenting structures across synthetic data, ex vivo lamb kidney datasets, and in vivo clinical partial nephrectomy surgery.
• Successfully fused prior knowledge from 3D CT scans with intraoperative video data for improved segmentation.
• The method effectively corrected camera calibration parameters within the segmentation process.

Conclusions:

• The proposed multi-modal approach significantly enhances endoscopic video segmentation for image-guided robotic surgery.
• This technique offers a robust solution for improving surgical perception and decision-making by integrating preoperative and intraoperative data.
• The method's ability to self-correct camera parameters adds to its reliability in real-world surgical scenarios.