Self-supervised multi-modal fusion network for multi-modal thyroid ultrasound image diagnosis

Area of Science:

• Medical Imaging
• Artificial Intelligence in Medicine
• Oncology

Background:

• Ultrasound is a common non-invasive method for detecting thyroid cancer.
• Limitations of ultrasound images necessitate additional diagnostic tools like shear wave elastography (SWE) and color Doppler ultrasound (CDUS).
• Current diagnostic processes are time-consuming, labor-intensive, and subjective, highlighting the need for automated solutions.

Purpose of the Study:

• To develop a deep learning-based multi-modal feature fusion network for the automatic diagnosis of thyroid disease.
• To improve the accuracy and efficiency of diagnosing benign versus malignant thyroid nodules.
• To reduce subjectivity and workload in clinical thyroid nodule diagnosis.

Main Methods:

• A deep learning network was designed to fuse data from three modalities: gray-scale ultrasound (US), SWE, and CDUS.
• Three ResNet18 models, initialized via self-supervised learning, served as branches for extracting features from each modality.
• A multi-modal multi-head attention branch was employed to refine common information, and a feature guidance module integrated inter-modal features.

Main Results:

• The proposed multi-modal network demonstrated effective fusion of diagnostic information from different ultrasound techniques.
• The method achieved fast and accurate assistance in diagnosing thyroid nodules on a self-collected dataset.
• The system showed potential for enhancing the diagnostic capabilities of sonographers.

Conclusions:

• The developed deep learning approach offers a promising solution for automated thyroid nodule diagnosis.
• Multi-modal data fusion significantly improves diagnostic accuracy and efficiency compared to single-modality approaches.
• This AI-powered tool can serve as valuable assistance for clinicians in thyroid cancer detection.