Generalizable stereo depth estimation with masked image modelling.

Area of Science:

• Computer Vision
• Medical Imaging
• 3D Reconstruction

Background:

• Accurate stereo depth estimation is crucial for 3D reconstruction, particularly in surgical applications.
• Supervised methods excel but struggle with limited surgical ground truth data, hindering generalizability.
• Self-supervised methods lack ground truth but face scale ambiguity and photometric inconsistencies.

Purpose of the Study:

• To develop a generalizable and high-performance stereo depth estimation method for 3D reconstruction.
• To overcome limitations of existing supervised and self-supervised approaches in surgical and natural scenes.
• To achieve state-of-the-art accuracy without direct training on target scene data.

Main Methods:

• A two-phase training procedure combining self-supervised masked image modeling (MIM) and supervised learning.
• Phase 1: Self-supervised representation learning using MIM to acquire generalizable semantic stereo features.
• Phase 2: Supervised learning on synthetic data using the MIM pre-trained model, incorporating perceptual losses to enhance stereo representations.

Main Results:

• The proposed method achieves sub-millimetre accuracy on surgical scenes and lowest errors on natural scenes.
• Demonstrates state-of-the-art performance in stereo depth estimation.
• Qualitative and quantitative evaluations confirm the approach's effectiveness and generalizability.

Conclusions:

• The two-phase training strategy effectively bridges the gap between self-supervised and supervised learning for stereo depth estimation.
• The method achieves high accuracy and generalizability without requiring direct training on specific scene data like surgical or natural images.
• This approach sets a new benchmark for stereo depth estimation in demanding applications such as robotic surgery.