Multimodal Continual Learning with Sonographer Eye-Tracking in Fetal Ultrasound

Area of Science:

• Medical image analysis
• Deep learning
• Artificial intelligence in healthcare

Background:

• Deep networks excel in medical imaging with large datasets.
• Incremental learning is challenging due to performance degradation on old classes when learning new ones.
• Mitigating 'forgetting' is crucial for evolving deployed medical AI algorithms.

Purpose of the Study:

• To explore expert knowledge's role in preventing forgetting in deep medical image analysis.
• To introduce a novel framework for mitigating forgetting in deep networks.
• To enhance the adaptability of medical AI models over time.

Main Methods:

• Developing a novel framework for incremental learning in medical image analysis.
• Integrating ultrasound video data with point-of-gaze tracking from expert sonographers.
• Employing a weighted distillation strategy to manage class imbalance effects.

Main Results:

• Demonstrated a novel framework to mitigate forgetting in deep learning models for medical imaging.
• Successfully combined ultrasound video and expert gaze data for training.
• Utilized weighted distillation to reduce class imbalance impact.

Conclusions:

• The proposed framework enhances deep networks' immunity to forgetting in medical image analysis.
• Expert sonographer gaze data offers a valuable form of knowledge for incremental learning.
• The approach supports the development of continuously learning and evolving medical AI systems.