Addressing Imbalanced Modal Incompleteness in Realistic Multi-Modal Medical Image Segmentation via Hierarchical...

Area of Science:

• Medical Image Analysis
• Machine Learning
• Computer Vision

Background:

• Multi-modal learning shows promise for medical image segmentation.
• Real-world data often suffers from modal incompleteness due to diverse sources.
• Existing methods struggle with training data limitations and model imbalance.

Purpose of the Study:

• To formulate and address the challenge of incomplete multi-modal learning in medical image segmentation.
• To propose Hierarchical Gradient Alignment (HGA) for balancing uni- and multi-modal data during training.
• To improve model fairness, robustness, and performance in the presence of missing modalities.

Main Methods:

• Hierarchical Gradient Alignment (HGA) is proposed, utilizing sequential meta-learning for multi-modal combinations and multi-level self-distillation for uni-modal data.
• Gradient direction alignment is achieved through meta-learning and self-distillation.
• Gradient magnitude alignment is performed using relative preference estimation to balance modal dominance.

Main Results:

• HGA consistently outperforms state-of-the-art methods for incomplete and imbalanced multi-modal learning.
• Experiments on five public benchmarks (BraTS, MyoPS, MSSEG) validate the effectiveness of HGA.
• HGA functions as a plug-and-play module, enhancing performance across various backbones.

Conclusions:

• HGA effectively addresses the challenge of uni- and multi-modal imbalance in medical image segmentation.
• The proposed method offers a robust solution for scenarios with incomplete multi-modal data.
• HGA provides consistent performance improvements and is adaptable to different network architectures.