A Large Language Model Screening Tool to Target Patients for Best Practice Alerts: Development and Validation

Area of Science:

• Artificial Intelligence in Medicine
• Clinical Decision Support Systems
• Natural Language Processing

Background:

• Best Practice Alerts (BPAs) are electronic health record messages to guide physician practice and resource utilization.
• Current BPAs are often non-selective, alerting broad patient populations.
• Large Language Models (LLMs) offer potential for selective patient identification for BPAs.

Purpose of the Study:

• To demonstrate an LLM-based screening tool for identifying patients suitable for deep vein thrombosis (DVT) anticoagulation prophylaxis BPAs.
• To develop an AI tool that excludes patients with active bleeding from DVT prophylaxis alerts.

Main Methods:

• A BioMed-RoBERTa model was fine-tuned on 500 MIMIC-III patient notes to classify physician notes for active bleeding.
• The model identified patients without active bleeding as appropriate candidates for DVT prophylaxis BPAs.
• A test set of 300 patient notes was used to evaluate the AI screening tool's performance.

Main Results:

• The AI screening tool achieved a precision-recall AUC of 0.82 and ROC AUC of 0.89.
• The tool reduced the number of alerts by 20% and increased alert applicability by 14.8%.
• Out of 300 patients, 72 had bleeding and 228 were appropriate for the DVT prophylaxis BPA.

Conclusions:

• LLMs can serve as effective screening tools for BPAs, demonstrating proof of concept.
• A HIPAA-compliant BioMed-RoBERTa model was deployed with minimal computational resources.
• Future LLMs are expected to significantly advance quality improvement initiatives in hospital medicine.