The Best of Both Worlds: How Combining a Large Language Model and a Rule-Based Algorithm Makes Catheter-Associated...

Area of Science:

• Infectious Disease Surveillance
• Artificial Intelligence in Healthcare
• Clinical Informatics

Background:

• Catheter-associated urinary tract infection (CAUTI) surveillance is vital for patient safety.
• Current electronic algorithms require manual chart review for CAUTI case confirmation.
• Large language models (LLMs) offer potential to enhance CAUTI surveillance.

Purpose of the Study:

• To evaluate the effectiveness of a large language model (LLM) in improving CAUTI surveillance.
• To compare different methods of applying the NHSN CAUTI definition using LLMs.
• To assess the potential of LLMs to reduce manual chart review burden.

Main Methods:

• Analysis of 919 potential CAUTI cases flagged by electronic surveillance.
• Comparison of rule-based logic with LLM incorporating Clinical Entity Augmented Retrieval (CLEAR).
• Application of National Healthcare Safety Network (NHSN) CAUTI definition to electronic medical record (EMR) data.

Main Results:

• The combined LLM and CLEAR approach achieved 90.0% sensitivity and 93.5% specificity.
• Expert adjudication improved sensitivity to 93.6% and specificity to 98.6%.
• False negatives were often due to missing symptom information in clinical documentation.

Conclusions:

• LLM augmentation significantly enhances CAUTI surveillance accuracy.
• This approach may improve efficiency by reducing manual chart review for infection preventionists.
• Optimizing clinical information presented to LLMs can further improve performance.