The Best of Both Worlds: How Combining a Large Language Model and a Rule-Based Algorithm Makes Catheter-Associated...

Catheter-associated urinary tract infection (CAUTI) surveillance is critical for patient safety. Many healthcare systems use electronic algorithms to flag candidate infection events but still require manual chart review to confirm CAUTI cases. We evaluated the use of a large language model (LLM) to enhance CAUTI surveillance.

We analyzed 919 potential CAUTI cases flagged by the electronic surveillance algorithm at Barnes-Jewish Hospital from 2021 to 2024. These cases were previously classified as 291 CAUTIs and 628 non-CAUTIs by trained infection preventionists (IPs). Several approaches for applying the National Healthcare Safety Network (NHSN) CAUTI definition after extracting clinical data from the electronic medical record (EMR) were compared.

Most patients were female (61.6%) with a median age 68 years (IQR 58, 77). Combining rule-based logic with Clinical Entity Augmented Retrieval (CLEAR) input into an LLM achieved the highest sensitivity (90.0%) and specificity (93.5%). Adjustment of false negatives and false positives after expert adjudication showed sensitivity of 93.6% and specificity of 98.6%. Chart review of false negatives revealed that disagreement with the gold standard mainly occurred due to missing symptom information in clinical documentation provided to the model.

Augmenting the existing algorithmic approach with LLM capabilities significantly enhances CAUTI surveillance and may improve efficiency by reducing the amount of time IPs spend performing manual chart review. Further improvements could be made by optimizing the clinical information presented to the model.