Detection of Pneumothorax with Deep Learning Models: Learning From Radiologist Labels vs Natural Language Processing...

Area of Science:

• Medical Imaging
• Artificial Intelligence
• Radiology

Background:

• Pneumothorax detection in chest radiographs is crucial for patient care.
• Deep learning models offer potential for automated detection.
• Labeling datasets for training these models can be time-consuming and resource-intensive.

Purpose of the Study:

• To compare the performance of deep learning models for pneumothorax detection.
• To evaluate models trained with radiologist-annotated labels versus natural language processing (NLP)-derived labels.
• To assess model generalizability using internal and external test sets.

Main Methods:

• Utilized the NIH ChestX-ray14 dataset, comprising over 112,000 chest radiographs.
• Created two datasets: one with NLP-derived pneumothorax labels and another with radiologist-confirmed labels.
• Trained three convolutional neural network (CNN) architectures (ResNet-50, DenseNet-121, EfficientNetB3) independently on both datasets.
• Evaluated model performance using the area under the receiver operating characteristic curve (AUC) on both the NIH internal test set and an external emergency department test set.

Main Results:

• CNN models trained with radiologist labels consistently achieved significantly higher AUCs compared to those trained with NLP labels across all architectures on both internal and external test sets.
• For instance, on the external test set, AUCs for radiologist-trained models ranged from 0.806 to 0.915, outperforming NLP-trained models (AUCs 0.686 to 0.822).
• The most significant performance improvement was observed with the EfficientNetB3 architecture, demonstrating the impact of label quality on model efficacy.

Conclusions:

• Radiologist-annotated labels lead to superior performance and generalizability of deep learning models for pneumothorax detection compared to NLP-derived labels.
• The findings underscore the importance of high-quality, expert-annotated data in developing robust medical AI tools.
• These results suggest that investing in radiologist-curated datasets can enhance the reliability and clinical utility of AI-powered diagnostic systems.