Related Experiment Video
Updated: Sep 20, 2025
05:12
Robotized Testing of Camera Positions to Determine Ideal Configuration for Stereo 3D Visualization of Open-Heart Surgery
Published on: August 12, 2021
2.2K
MSDESIS: Multitask Stereo Disparity Estimation and Surgical Instrument Segmentation
IEEE Transactions on Medical Imaging
|June 8, 2022
Summary
This study introduces a novel deep learning framework for surgical robotics, jointly estimating 3D geometry and instrument segmentation. The multi-task approach enhances accuracy and enables real-time performance for improved surgical navigation.
Area of Science:
- Computer Vision
- Medical Robotics
- Machine Learning
Background:
- Surgical navigation and robotic surgery require accurate 3D reconstruction and instrument detection.
- Existing methods often address these tasks in isolation, neglecting their inherent relationship.
Purpose of the Study:
- To develop a unified learning-based framework for joint disparity estimation and tool segmentation.
- To leverage shared features for improved performance in both tasks.
Main Methods:
- A novel deep learning architecture with a shared feature encoder for joint disparity and segmentation estimation.
- Training and evaluation of network variants using multi-task and single-task learning schemes.
- Implementation of a domain adaptation strategy using monocular segmentation data.
Main Results:
- Supervising the segmentation task demonstrably improved disparity estimation accuracy.
- Domain adaptation reduced disparity End-Point-Error by 77.73% and depth mean absolute error by 61.73%.
- The best multi-task model achieved 89.15% mIoU for segmentation and 3.18 mm MAE for depth.
Conclusions:
- The proposed multi-task framework effectively integrates 3D reconstruction and instrument detection.
- The model achieves real-time performance, processing high-resolution stereo input at 22 FPS.
- This approach offers significant advancements for surgical navigation and robotic automation.
