Large language model-based uncertainty-adjusted label extraction for artificial intelligence model development in...

Area of Science:

• Artificial Intelligence in Radiology
• Machine Learning for Medical Imaging
• Natural Language Processing in Healthcare

Background:

• Radiology reports contain valuable diagnostic information.
• Extracting structured data from free-text reports is challenging.
• AI models require large, accurately labeled datasets for training.

Purpose of the Study:

• To assess GPT-4o's zero-shot capability in extracting structured diagnostic labels from radiology reports.
• To evaluate the impact of these extracted labels, including uncertainty, on multi-label image classification performance for musculoskeletal radiographs.

Main Methods:

• Retrospective analysis of clavicle, elbow, and thumb radiographs.
• GPT-4o extracted labels (present, absent, uncertain) from anonymized reports.
• Uncertainty labels were handled inclusively and exclusively for model training.
• ResNet50 architecture was used for multi-label classification.
• Performance was validated on internal and external test sets using AUC and other metrics.

Main Results:

• GPT-4o achieved >98% accuracy in automatic label extraction across test sets.
• Label-based models demonstrated competitive performance (e.g., elbow AUC=0.80) regardless of uncertainty handling.
• Models generalized well to external datasets with consistent performance.
• No significant performance differences were observed across labeling strategies or datasets (p≥0.15).

Conclusions:

• GPT-4o effectively extracts high-accuracy diagnostic labels from radiology reports.
• These labels facilitate the training of competitive multi-label image classification models.
• Handling of uncertainty in extracted labels did not significantly affect model performance.