Evaluating a large language model's accuracy in chest X-ray interpretation for acute thoracic conditions

Area of Science:

• Artificial Intelligence in Healthcare
• Medical Imaging Analysis
• Emergency Radiology

Background:

• Artificial intelligence (AI) is rapidly advancing, with significant potential to transform healthcare delivery.
• Chest X-rays are critical for diagnosing acute thoracic conditions in the emergency department (ED).
• Interpretation delays of chest X-rays can impede timely clinical decision-making; AI models offer potential diagnostic support, but large language models (LLMs) in emergency radiology are under-explored.

Purpose of the Study:

• To assess the feasibility of ChatGPT in interpreting chest X-rays for common acute thoracic conditions in the ED.
• To evaluate the diagnostic performance of ChatGPT 4.0 with the "X-Ray Interpreter" add-on across seven distinct pathology categories.

Main Methods:

• Analysis of 1400 chest X-ray images from the NIH Chest X-ray dataset.
• Categorization of images into seven pathologies: Atelectasis, Effusion, Emphysema, Pneumothorax, Pneumonia, Mass, and No Finding.
• Evaluation of ChatGPT 4.0's diagnostic performance using the "X-Ray Interpreter" add-on.

Main Results:

• ChatGPT achieved high performance in identifying normal chest X-rays (sensitivity 98.9%, specificity 93.9%, accuracy 94.7%).
• Diagnostic performance varied across pathologies; best results were for pneumonia (sensitivity 76.2%, specificity 93.7%) and pneumothorax (sensitivity 77.4%, specificity 89.1%).
• Lower performance was noted for detecting atelectasis and emphysema.

Conclusions:

• ChatGPT shows potential as an adjunct tool for distinguishing normal from abnormal chest X-rays, particularly for conditions like pneumonia.
• Diagnostic accuracy for subtle conditions requires enhancement.
• Future research should explore integrating ChatGPT with specialized image recognition models to improve performance in medical imaging and education.