Perseus: perception with semantic endoscopic understanding and SLAM

Area of Science:

• Medical Robotics
• Computer Vision
• Surgical Navigation

Background:

• Natural orifice surgery offers minimally invasive benefits but presents visualization challenges.
• Enhanced scene understanding is crucial for surgeon guidance and automation in these procedures.

Purpose of the Study:

• To develop a perception pipeline for semantically informed 3D reconstructions in natural orifice surgery.
• To improve surgeon guidance and enable automation through reliable surgical scene understanding.

Main Methods:

• Integrated learning-based segmentation, depth estimation, and real-time SLAM for 3D reconstruction from monocular videos.
• Propagated 2D semantic labels into 3D space to create real-time segmented surgical maps.
• Utilized robot pose registration to resolve scale ambiguity and enable real-time use in robotic surgery.

Main Results:

• Achieved sub-millimeter reconstruction accuracy and pose registration RMSE of 0.9 mm.
• Demonstrated improved processing time over offline methods while maintaining reconstruction accuracy.
• Showcased robustness in challenging scenarios like submerged prostate and cadaver airway explorations.

Conclusions:

• A modular perception pipeline integrating semantic segmentation with real-time monocular SLAM was presented for natural orifice surgeries.
• The pipeline provides a promising solution for surgical scene understanding, facilitating automation and surgeon guidance.
• The framework's plug-and-play design and generalizability support scalable clinical translation.