Can surgical computer vision benefit from large-scale visual foundation models?

Area of Science:

• Computer Vision
• Medical Imaging
• Surgical Technology

Background:

• Foundation models pretrained on diverse visual data are increasingly explored for specialized domains.
• Surgical computer vision tasks, such as instrument and uterus segmentation, require robust and accurate models.

Purpose of the Study:

• To investigate the efficacy of foundation models for surgical computer vision.
• To develop and evaluate novel adaptation methods for these models in instrument and uterus segmentation tasks.

Main Methods:

• Utilized DINOv1, DINOv2, and SAM backbones with ART-Net and SurgAI3.8K datasets.
• Implemented supervised, unsupervised, and few-shot learning adaptations, including DINO-UNet and SAM adaptations.
• Evaluated 17 instrument and 7 uterus segmentation models against existing methods.

Main Results:

• Few-shot learning with a linear decoder proved feasible.
• Unsupervised and linear decoder methods showed utility in data-scarce settings.
• Proposed DPT and DINO-UNet adaptations achieved new state-of-the-art performance, outperforming previous bests by significant margins (e.g., 5.6 pp for instrument segmentation).

Conclusions:

• Visual foundation models, specifically DINO and SAM, hold substantial promise for surgical computer vision.
• These models are particularly valuable in medical image analysis scenarios characterized by limited or complex data.