Inconsistent Performance of Deep Learning Models on Mammogram Classification.

Area of Science:

• Artificial Intelligence in Medical Imaging
• Deep Learning for Mammography Analysis
• Radiology and Diagnostic Imaging

Background:

• Deep learning models achieve high performance in mammogram image classification, sometimes surpassing radiologists.
• Concerns exist regarding the consistency and generalization capabilities of these models on external, unseen data.
• Assessing transferability of deep learning performance across diverse mammogram datasets is crucial for clinical adoption.

Purpose of the Study:

• To evaluate the performance consistency of deep learning models on external mammogram datasets.
• To determine if high performance on a training dataset transfers to data with different distributions.
• To assess the generalizability of deep learning models in mammography.

Main Methods:

• Six deep learning models (three published, three novel) were evaluated.
• Models were trained/validated on Digital Database for Screening Mammography (DDSM) or combined datasets.
• Performance was tested on three external datasets (INbreast, MIAS, UKy) using area under the receiver operating characteristic curve (auROC).

Main Results:

• Published models achieved high auROC (0.88-0.95) on validation data.
• Novel models achieved moderate auROC (0.71-0.79) on validation data.
• All models showed significantly decreased performance on external test datasets (auROC 0.44-0.65).

Conclusions:

• Deep learning model performance on mammograms is inconsistent across datasets.
• High performance on one dataset does not readily transfer to unseen external data.
• Models require further rigorous assessment and validation before clinical implementation.