Large Language Models Perform at Chance Level in the Diagnosis of Pediatric Pneumonia Using Chest Radiographs.

Area of Science:

• Artificial Intelligence in Medicine
• Pediatric Radiology
• Medical Diagnostics

Background:

• Pneumonia is a major global child health concern.
• Chest radiographs (CXRs) are crucial for diagnosing pediatric pneumonia.
• Differentiating bacterial from viral pneumonia on CXRs is challenging.

Purpose of the Study:

• To evaluate the diagnostic performance of general-purpose large language models (LLMs) in identifying pediatric pneumonia on CXRs.
• To assess the reliability of LLMs in distinguishing between bacterial pneumonia, viral pneumonia, and normal CXRs.
• To understand the limitations of current LLMs for unsupervised medical image interpretation.

Main Methods:

• Four publicly available LLMs (ChatGPT, Claude, Gemini, Grok) were tested on 44 pediatric CXRs.
• Images were classified as bacterial pneumonia, viral pneumonia, or normal.
• Each LLM interpreted each image twice; accuracy and internal consistency were measured against human expert consensus.

Main Results:

• Average diagnostic accuracy across all LLMs was 31%, equivalent to chance.
• Accuracy was highest for viral pneumonia (54%) and lowest for normal CXRs (18%).
• Internal consistency ranged from 46% to 71%, indicating unreliable performance; concordance with experts did not exceed 49%.

Conclusions:

• General-purpose LLMs are currently unreliable for diagnosing pediatric pneumonia on CXRs.
• Their low accuracy, especially in ruling out disease, and lack of internal consistency pose risks for unsupervised clinical deployment.
• Future AI tools must be purpose-built, trained on diverse data, and integrated with clinical oversight for safe use.