Subsampled randomized Fourier GaLore for adapting foundation models in depth-driven liver landmark segmentation

Area of Science:

• Medical Imaging
• Computer-Assisted Surgery
• Artificial Intelligence in Medicine

Background:

• Accurate anatomical structure detection is crucial for computer-assisted interventions, especially in laparoscopic surgery.
• 2D video limitations in depth perception and landmark localization pose significant challenges.
• Existing methods struggle with fusing RGB and depth features and adapting large vision models to surgical data.

Purpose of the Study:

• To develop a depth-guided segmentation framework for precise anatomical structure delineation in laparoscopic liver surgery.
• To address challenges in fusing RGB and depth data and efficiently adapting foundation models for surgical applications.
• To improve landmark localization and depth perception in 2D surgical video streams.

Main Methods:

• Proposed a dual-encoder framework using SAM2 (RGB) and Depth Anything V2 (depth) for semantic and geometric cues.
• Introduced SRFT-GaLore for efficient fine-tuning of SAM2's high-dimensional attention layers.
• Implemented a cross-attention fusion module to integrate RGB and depth features.
• Validated cross-dataset generalization on L3D and LLSD datasets.

Main Results:

• Achieved 4.85% improvement in Dice Similarity Coefficient (DSC) and 11.78-point reduction in Average Symmetric Surface Distance (ASSD) on the L3D dataset.
• Demonstrated strong cross-dataset robustness and adaptability on the LLSD dataset.
• Outperformed SAM-based baselines, indicating effective generalization to unseen surgical environments.

Conclusions:

• The SRFT-GaLore-enhanced dual-encoder framework enables scalable, precise segmentation in depth-constrained surgical settings.
• Foundation model adaptation shows potential for real-time computer-assisted interventions.
• Future work will explore transformer-based decoders for deeper cross-modal fusion and improved performance.