Can the generalizability issue of artificial intelligence be overcome? Pneumothorax detection algorithm

Area of Science:

• Medical Imaging
• Artificial Intelligence in Medicine
• Radiology

Background:

• Generalizability of artificial intelligence (AI) models is a critical challenge in clinical applications.
• AI models trained on data from a single institution may not perform well in different healthcare settings.
• Developing robust AI for accurate pneumothorax detection is essential for timely patient management.

Purpose of the Study:

• To enhance the generalizability of a deep learning-based pneumothorax detection algorithm (PDA-Alpha) through external validation.
• To assess the impact of external validation using a small dataset on the performance of the AI model.
• To compare the diagnostic performance of the validated AI model against human physicians.

Main Methods:

• A deep learning algorithm (PDA-Alpha) was developed using chest radiographs from Ankara University Faculty of Medicine (AUFM).
• External validation was performed by developing PDA-Beta using a small dataset from Health Sciences University (HSU).
• Both PDA-Alpha and PDA-Beta were evaluated on a test dataset from HSU, and results were compared with physician performance.

Main Results:

• PDA-Alpha and PDA-Beta achieved high areas under the curve (0.993 and 0.986, respectively).
• External validation significantly improved the positive predictive value from 0.525 to 0.886 (p=0.041).
• PDA-Beta demonstrated improved performance with fewer false positives compared to PDA-Alpha, outperforming physicians in sensitivity and specificity.

Conclusions:

• External validation, even with a small dataset, can significantly enhance the generalizability and performance of AI models for pneumothorax detection.
• The validated AI model shows promise for reliable and accurate pneumothorax diagnosis in diverse clinical settings.
• Further research is needed to determine the optimal size of external validation datasets for AI model generalizability.