Multi-modality multiorgan image segmentation using continual learning with enhanced hard attention to the task.

Area of Science:

• Artificial Intelligence
• Medical Image Analysis
• Machine Learning

Background:

• Deep neural networks (DNNs) can mimic human brain functions through continual learning, enabling them to learn multiple tasks sequentially.
• Current continual learning methods for medical image segmentation are limited to single-modality images and specific anatomical locations.

Purpose of the Study:

• To introduce and assess an advanced continual learning technique, enhanced hard attention to the task (eHAT), for DNN-based multi-modality and multiorgan segmentation.
• To evaluate eHAT's performance in complex medical imaging scenarios.

Main Methods:

• Utilized four public datasets (lumbar spine CT/MRI, heart MRI, brain MRI) for segmenting vertebral bodies, right ventricle, and brain tumors.
• Tested eHAT on three-task and four-task models, comparing its performance against state-of-the-art continual learning methods.
• Quantified multitask performance using forgetting rate (difference in Dice coefficients and Hausdorff distances) and backward transfer (BWT).

Main Results:

• eHAT demonstrated substantially improved forgetting rates (-2.51% to -0.60% for three tasks, -2.54% to -1.59% for four tasks) compared to the original HAT (-18.13% to -3.59%).
• Four-task U-net models with eHAT achieved comparable performance with significantly fewer parameters (half the channels).
• eHAT exhibited significantly superior BWT (-3% to 0%) compared to HAT (-22% to -4%), indicating better knowledge transfer.

Conclusions:

• eHAT is the first method to successfully achieve continual learning for multi-modality, multiorgan segmentation using a single DNN.
• The proposed eHAT method offers improved forgetting rates and enhanced knowledge transfer capabilities in continual learning for medical imaging.