External Validation of Deep Learning Algorithms for Radiologic Diagnosis: A Systematic Review

Area of Science:

• Radiology
• Artificial Intelligence
• Medical Informatics

Background:

• Deep learning (DL) algorithms are increasingly used for radiologic diagnosis.
• Assessing the generalizability of these algorithms is crucial for clinical adoption.

Purpose of the Study:

• To systematically review and assess the generalizability of published deep learning algorithms for image-based radiologic diagnosis.
• To evaluate the performance decrease of DL algorithms when applied to external datasets.

Main Methods:

• Systematic review of PubMed-indexed studies (Jan 2015-Apr 2021) on DL algorithms for radiologic diagnosis with external validation.
• Exclusion of studies using nonimaging features or non-DL methods.
• Extraction of internal and external performance measures and study characteristics.

Main Results:

• Eighty-three studies (86 algorithms) were included.
• 81% of algorithms showed decreased external performance compared to internal performance.
• 49% had at least a modest decrease (≥0.05), and 24% had a substantial decrease (≥0.10).

Conclusions:

• The majority of published DL algorithms for radiologic diagnosis exhibit diminished performance on external datasets.
• A significant portion of algorithms experience substantial performance decreases, indicating generalizability issues.
• No specific study characteristics were found to predict the performance difference between internal and external validation.