Integrating visual and language cues via state space models for medical image segmentation

Area of Science:

• Medical Image Analysis
• Artificial Intelligence
• Computer Vision

Background:

• Medical image segmentation is crucial but challenged by low contrast, ambiguous boundaries, and limited annotated data.
• Integrating clinical text prompts offers a solution, but modeling cross-modal dependencies is difficult for current deep learning models.

Purpose of the Study:

• To develop a novel neural framework for reliable medical image segmentation guided by language prompts.
• To address the limitations of current deep learning architectures in handling multi-modal dependencies and prediction uncertainty.

Main Methods:

• Introduced a Multimodal Interactive Guide Decoder (MIGD) using State Space Models (SSMs) for efficient global context capture and cross-attention for feature alignment.
• Proposed a Multi-Expert Uncertainty Refinement (MEUR) module, using Choquet integrals to aggregate expert opinions for pixel-wise uncertainty estimation.

Main Results:

• Achieved state-of-the-art or competitive performance on three benchmarks (QaTa-COVID19, MosMedData+, MoNuSeg) for radiology and histopathology tasks.
• Outperformed strong competitors (LViT, RecLMIS) in accuracy (Dice/mIoU) and computational efficiency (GFLOPs).
• Demonstrated superior prediction stability in challenging segmentation scenarios.

Conclusions:

• The proposed framework effectively leverages State Space Models for dynamic, selective multi-modal fusion in medical image segmentation.
• The MIGD and MEUR modules enhance segmentation accuracy, efficiency, and reliability, offering a promising solution for clinical applications.