A generalist foundation model and database for open-world medical image segmentation.

Area of Science:

• Artificial Intelligence
• Medical Imaging
• Computer Vision

Background:

• Vision foundation models offer versatile medical segmentation but suffer from negative transfer due to irrelevant pre-training data.
• Limited evaluation exists for these models in diverse open-world and out-of-distribution (OOD) medical scenarios.

Purpose of the Study:

• To develop a generalist medical segmentation model (MedSegX) robust to OOD challenges.
• To create a comprehensive medical segmentation database (MedSegDB) for training and evaluation.
• To assess MedSegX's performance across various medical segmentation tasks and settings.

Main Methods:

• Construction of MedSegDB, a hierarchical database from 129 public and 5 in-house medical segmentation datasets.
• Development of MedSegX, a vision foundation model utilizing Contextual Mixture of Adapter Experts (ConMoAE) for open-world segmentation.
• Comprehensive evaluation of MedSegX on in-distribution (ID), OOD, and real-world clinical segmentation tasks.

Main Results:

• MedSegX achieved state-of-the-art performance on various medical segmentation tasks in ID settings.
• MedSegX maintained strong performance in OOD and real-world clinical settings, excelling in zero-shot and data-efficient generalization.
• MedSegX outperformed existing foundation models in challenging generalization scenarios.

Conclusions:

• MedSegX represents a significant advancement in generalist medical segmentation, effectively addressing negative transfer and OOD limitations.
• The proposed ConMoAE approach enhances model adaptability for diverse medical imaging applications.
• MedSegDB provides a valuable resource for future research in medical foundation models.