SSIFNet: Spatial-temporal stereo information fusion network for self-supervised surgical video inpainting

Area of Science:

• Medical Imaging
• Computer Vision
• Robotic Surgery

Background:

• Minimally invasive robot-assisted surgery relies on stereo endoscopes for guidance.
• Restricted fields-of-view and instrument occlusions pose significant visualization challenges.
• Occluded anatomical landmarks hinder accurate surgical assessment.

Purpose of the Study:

• To propose a novel end-to-end trainable network, SSIFNet, for inpainting surgical videos.
• To address the challenge of instrument occlusions in robot-assisted endoscopic surgery.
• To enhance visualization by reconstructing occluded regions in surgical scenes.

Main Methods:

• Developed SSIFNet, featuring optical flow-guided deformable feature propagation (OFDFP), spatial-temporal stereo focal transformer (S²FT), and stereo-consistency enforcement (SE) modules.
• Employed self-supervised training with simulated occlusions using a novel loss function.
• Integrated flow completion, disparity matching, cross-warping consistency, warping-consistency, image, and adversarial loss terms.

Main Results:

• SSIFNet successfully inpaints occluded regions in surgical videos, maintaining spatial, temporal, and stereo-consistency.
• The network achieves high-fidelity and accurate occlusion reconstructions in both stereo views.
• Experimental results demonstrate SSIFNet's superiority over state-of-the-art video inpainting methods.

Conclusions:

• SSIFNet effectively overcomes visualization challenges caused by instrument occlusions in robot-assisted surgery.
• The proposed method enhances surgical scene understanding by providing consistent and accurate reconstructions.
• SSIFNet offers a promising solution for improving image guidance in minimally invasive procedures.