Dual-stage 3D medical image segmentation integrating learnable prompt generation and memory attention

Area of Science:

• Medical Imaging
• Computer Vision
• Artificial Intelligence

Background:

• Current 3D medical image segmentation methods like MedSAM2 struggle with automatic prompt generation, especially for rare cases or multi-object scenarios.
• Existing automatic prompt generation methods often fail to capture sufficient spatiotemporal context and semantic information from image features, leading to suboptimal segmentation performance.
• The lack of rich contextual and semantic information in automatically generated prompts hinders accurate 3D medical image segmentation.

Purpose of the Study:

• To develop a novel dual-stage 3D medical image segmentation network, DSSAM2-LAPG, that integrates a Learnable Automatic Prompt-space Generator (LAPG) with memory attention.
• To address the limitations of current automatic prompt generation in capturing 3D context, encoding rare-object semantics, and providing instance-aware guidance.
• To improve the performance of 3D medical image segmentation when relying solely on automatically generated prompts.

Main Methods:

• Proposed DSSAM2-LAPG, a dual-stage network featuring a Learnable Automatic Prompt-space Generator (LAPG) and memory attention for 3D medical image segmentation.
• The LAPG acts as a trainable mapper, transforming 3D image features into a semantically-rich, spatially-aligned prompt embedding space using learnable object tokens.
• A memory attention mechanism refines segmentation by integrating generated prompts with historical context from a support memory, ensuring 3D consistency.

Main Results:

• DSSAM2-LAPG demonstrated significant performance improvements over the MedSAM2 baseline across multiple datasets without manual prompts.
• Achieved Dice score improvements of 7.2% on the XYCH-cervical dataset, 6.0% on the CCTH-Cervical dataset, and 4.1% on the Multi-Organ BTCV dataset.
• The proposed method effectively overcomes limitations in capturing 3D context and handling rare or multi-object segmentation scenarios.

Conclusions:

• DSSAM2-LAPG offers a robust solution for automatic 3D medical image segmentation, outperforming existing methods by effectively generating semantically-rich and spatially-aligned prompts.
• The integration of LAPG and memory attention successfully addresses critical challenges in 3D medical image segmentation, including context capture and rare-object handling.
• The developed network provides a promising direction for advancing automated segmentation in complex medical imaging applications.