Achieving Expert-Level Clinical Infection Detection with LLMs from Clinical Documents: Validation in Complex Patient...

Area of Science:

• Hepatology and Infectious Diseases
• Artificial Intelligence in Medicine
• Clinical Informatics

Background:

• Systemic infections are a major cause of mortality in cirrhosis patients.
• Current infection identification methods (ICD-10, manual review) are unreliable and inefficient.
• Accurate infection detection is crucial for patient care and predictive modeling in cirrhosis.

Purpose of the Study:

• To develop and validate a large language model (LLM)-based approach for automated infection classification and subtyping.
• To improve the accuracy and efficiency of identifying infections in emergency department (ED) patients with cirrhosis.
• To overcome the limitations of existing methods for infection surveillance in this population.

Main Methods:

• Developed INFEHR (INfection identification and subtyping using Free-text EHR analysis), an LLM pipeline using Claude 3.5 Sonnet.
• Analyzed 72-hour clinical notes from 1,000 ED encounters of cirrhosis patients.
• Compared INFEHR performance against physician-adjudicated data, ICD-10 codes, and CDC sepsis criteria.

Main Results:

• INFEHR achieved 94.7% overall accuracy, 99.5% sensitivity, and 92.8% PPV for infection detection.
• The LLM significantly outperformed ICD-10 classification (p < 0.0001) across all metrics.
• INFEHR demonstrated high accuracy in pathogen type and infection site classification, processing notes rapidly.

Conclusions:

• INFEHR provides a scalable, reproducible, and accurate method for infection phenotyping in cirrhosis.
• The LLM overcomes limitations of traditional coding and manual review for high-throughput surveillance.
• INFEHR facilitates improved cohort construction for research and potential real-time clinical decision support.