Vision foundation model for 3D magnetic resonance imaging segmentation, classification, and registration.

Area of Science:

• Artificial Intelligence
• Medical Imaging
• Computer Vision

Background:

• Vision foundation models (VFMs) offer general representations for downstream tasks.
• Existing VFMs often pre-trained on non-MRI modalities, limiting MRI application performance.
• Differences in imaging principles and data distribution hinder VFM versatility in MRI.

Purpose of the Study:

• Introduce Triad, a VFM specifically designed for 3D MRI segmentation, classification, and registration.
• Leverage the largest 3D MRI pre-training dataset (Triad-129K) for robust representation learning.
• Enhance VFM performance and versatility in 3D MRI applications.

Main Methods:

• Developed Triad using the SimMIM framework on 129K 3D MRI volumes.
• Constrained semantic distribution using textual descriptions of modality and imaging parameters.
• Evaluated Triad on 25 downstream datasets across segmentation, classification, and registration tasks.

Main Results:

• nnUNet-Triad-SimMIM improved segmentation by 2.13% over nnUNet-Scratch on 17 datasets.
• Swin-B-Triad-SimMIM achieved 4.38% improvement in classification over Swin-B-Scratch on 5 datasets.
• SwinUNETR-Triad-SimMIM improved registration by 3.84% over SwinUNETR-Scratch on 2 datasets.

Conclusions:

• Large-scale pre-training on 3D MRI data significantly boosts performance in downstream tasks.
• Triad demonstrates the value of MRI-specific VFMs for medical imaging applications.
• Consistent data modalities and organ characteristics between pre-training and downstream tasks are key for performance gains.