Comparative Evaluation of Clinical Large Language Models and Machine Learning to Predict Antimicrobial Resistance in...

Area of Science:

• Medical Informatics
• Computational Biology
• Infectious Diseases

Background:

• Antimicrobial resistance (AMR) poses a significant threat, particularly in critically ill patients.
• Effective empiric antimicrobial therapy guidance is crucial for managing infections, especially with rising AMR rates.
• Predictive models for AMR are needed for clinical adoption, requiring scalability and generalizability.

Purpose of the Study:

• To compare the performance of a clinical large language model (LLM) against traditional machine learning (ML) for predicting AMR and methicillin-resistant Staphylococcus aureus (MRSA).
• To evaluate the utility of electronic health record (EHR) data at illness onset for early AMR and MRSA prediction in hospital-onset sepsis.
• To assess the potential of LLMs in guiding antimicrobial therapy in the context of prevalent AMR.

Main Methods:

• Utilized a publicly available clinical LLM (Gatortron) and traditional ML models.
• Analyzed EHR data from approximately 150,000 hospitalizations (2010-2023) at a large tertiary care system.
• Focused on a hospital-onset sepsis cohort (2,019 encounters) to predict AMR and MRSA presence.

Main Results:

• LLMs demonstrated superior performance in predicting MRSA compared to traditional ML models (AUC 0.73 vs. 0.66).
• LLMs achieved significantly higher F1 scores for MRSA prediction (0.43 vs. 0.16 for ML).
• The LLM achieved a negative predictive value of at least 90% for MRSA prediction across most infection types.

Conclusions:

• Clinical LLMs show significant potential for early prediction of AMR and MRSA using EHR data.
• LLMs offer a promising approach to guide empiric antimicrobial therapy, even with simplified feature sets.
• Further refinement of LLM-based prediction models is necessary to enhance sensitivity and clinical applicability for widespread adoption.