Reducing Annotation Burden Through Multimodal Learning.

Area of Science:

• Medical imaging
• Machine learning
• Data science

Background:

• Machine learning with multimodal data requires optimal data fusion techniques.
• Deep learning offers advanced methods for combining diverse data types.

Purpose of the Study:

• To compare deep learning-based multimodal fusion techniques (Early, Late, Model) for classifying radiological images and text reports.
• To assess multimodal vs. unimodal learning performance.
• To investigate data requirements for comparable performance.

Main Methods:

• Comparison of Early, Late, and Model fusion techniques.
• Evaluation of multimodal versus unimodal learning.
• Analysis of labeled data quantity impact on model performance.

Main Results:

• Multimodal fusion methods, particularly Early fusion, yield competitive classification results using less labeled training data than unimodal methods.
• Unimodal models achieve comparable performance to multimodal models with sufficient training data.
• Early fusion demonstrated the greatest reduction in required labeled data.

Conclusions:

• Deep learning multimodal fusion can significantly reduce the need for labeled training data in medical image and text analysis.
• This reduction lessens the annotation burden on domain experts.
• Multimodal learning presents a viable strategy for efficient model development in healthcare AI.