Fusing Tool Segmentation Predictions from Pose-Informed Morphological Polar Transform of Endoscopic Images

Area of Science:

• Medical Image Analysis
• Computer Vision
• Surgical Robotics

Background:

• Accurate surgical tool segmentation is crucial for robot-assisted minimally invasive surgery (RMIS).
• Existing deep learning fusion methods require large datasets and can lack explainability.
• Lack of reliable force and tactile feedback remains a challenge in RMIS.

Purpose of the Study:

• To develop and evaluate novel methods for fusing semantic image segmentation predictions.
• To combine spatial frequency and edge features for improved surgical tool segmentation.
• To enhance explainability in segmentation fusion for RMIS applications.

Main Methods:

• Utilized a U-Net architecture for segmentation on tool-labeled sinus surgery endoscopy images.
• Employed morphological polar transform for pre-processing, generating two predictions (tool-tip and vanishing point).
• Proposed three low-level feature-based fusion approaches: gradient estimation, Laplacian pyramid, and modified spatial frequency.

Main Results:

• The Laplacian pyramid and modified spatial frequency methods demonstrated enhanced segmentation compared to individual predictions.
• Explored explainability using unsupervised clustering and a ResNet-18 model to identify optimal prediction pairs for fusion.
• Investigated the potential of fused predictions in domains beyond surgical tool segmentation.

Conclusions:

• Low-level feature fusion methods offer a viable alternative to deep learning approaches for segmentation fusion.
• Enhanced segmentation accuracy can potentially improve vision-based force estimation in RMIS.
• The proposed fusion techniques show promise for broader applications in image analysis.