Performance of Open-Source LLMs in Identifying Pediatric Pneumonia From Free-Text Chest Radiograph Reports.

Area of Science:

• Medical Informatics
• Artificial Intelligence in Medicine
• Pediatric Radiology

Background:

• Community-acquired pneumonia is a common pediatric illness.
• Accurate interpretation of chest radiographs (CXRs) is crucial for diagnosis.
• Automated analysis of CXR reports can improve efficiency and consistency.

Purpose of the Study:

• To develop and validate an automated system for classifying pediatric CXR reports for pneumonia.
• To evaluate the performance of open-source large language models (LLMs) for this task.

Main Methods:

• Retrospective study of 1000 pediatric emergency department CXR reports.
• Reports adjudicated by two physicians for pneumonia presence (positive, negative, indeterminate).
• Evaluation of five open-source LLMs using a 70/30 train-test split, assessing both three-class and binary classification.

Main Results:

• Gemma2 9B achieved the highest performance, with F1 scores of 0.82 (pneumonia) and 0.97 (no pneumonia) in three-class classification.
• Binary classification further improved performance (F1=0.97 for Gemma2 9B).
• LLMs significantly outperformed traditional NLP classifiers; discrepancies often due to report ambiguity.

Conclusions:

• Open-source LLMs demonstrate high accuracy in classifying pediatric CXR reports for pneumonia.
• LLM integration is feasible for decision support and quality improvement in pediatric emergency care.
• This technology can enhance radiographic interpretation and patient care.