Evaluation of large language models for antimicrobial classification: implications for antimicrobial stewardship...

Area of Science:

• Artificial Intelligence in Healthcare
• Pharmacology and Drug Classification
• Antimicrobial Stewardship

Background:

• Accurate identification of antimicrobial medications is crucial for effective antimicrobial stewardship.
• Manual classification of large medication datasets is time-consuming and prone to errors.
• Large language models (LLMs) offer potential for automating complex classification tasks.

Purpose of the Study:

• To assess the efficacy of LLMs in classifying medications as antimicrobial or non-antimicrobial.
• To evaluate the impact of targeted feedback on LLM classification accuracy.
• To determine the utility of LLMs in supporting antimicrobial stewardship programs.

Main Methods:

• A dataset of 7,239 unique medication entries was classified by four LLMs (ChatGPT-3.5, Copilot GPT-4o, Claude Sonnet 4, Gemini 2.5 Flash).
• Models underwent a two-phase evaluation: initial unguided classification followed by feedback-informed reclassification on misclassified cases.
• Performance metrics included accuracy, macro F1-score, processing time, and error reduction rates (ERRs).

Main Results:

• Post-feedback, Gemini achieved 99.6% accuracy and Claude Sonnet 4 achieved 99.4% accuracy.
• Gemini demonstrated the highest macro-F1 score (98.9%) and ERR (69.2%).
• Processing times varied significantly, with Copilot and ChatGPT-3.5 being the fastest.

Conclusions:

• High-performing LLMs can achieve accuracy levels suitable for automating initial antimicrobial classification within stewardship workflows.
• Variability in LLM performance necessitates careful model selection and ongoing human oversight for clinical applications.
• LLMs, particularly Gemini and Claude, show significant potential to enhance efficiency and accuracy in antimicrobial stewardship.