Zero-shot large language model application for surgical site infection auditing

Area of Science:

• Medical Informatics
• Artificial Intelligence in Healthcare
• Clinical Surveillance

Background:

• Surgical site infections (SSI) pose a significant clinical challenge.
• Artificial intelligence (AI), specifically large language models (LLMs), offers potential for enhanced SSI monitoring.
• Current monitoring methods can be labor-intensive and may benefit from AI assistance.

Purpose of the Study:

• To evaluate the feasibility and performance of a large language model (LLM) in identifying surgical site infections (SSI) from clinical notes.
• To assess the accuracy, sensitivity, and specificity of the LLM in detecting SSI cases.
• To determine the timeliness of SSI detection by the LLM.

Main Methods:

• A retrospective study utilizing the Llama 3.0 70-billion parameter model.
• Analysis of clinical inpatient and outpatient progress notes from 28 patients (14 SSI cases, 14 controls).
• LLM classification of notes for SSI presence, followed by performance characteristic analysis.

Main Results:

• The LLM achieved 93% overall patient-level accuracy in identifying SSI.
• Demonstrated 100% sensitivity and 86% specificity for SSI detection.
• For initial case identification, 92.3% of notes flagged by the LLM were on or before the clinician-identified infection onset date.

Conclusions:

• Large language models (LLMs) are a feasible tool for screening medical records to detect surgical site infections (SSI).
• The study highlights the potential of LLMs to assist in clinical workflows for infection surveillance.
• Further research is recommended to explore LLM integration into routine clinical practice for SSI management.