Automated Identification of Stroke Thrombolysis Contraindications from Synthetic Clinical Notes: A Proof-of-Concept...

Area of Science:

• Artificial Intelligence in Medicine
• Computational Neuroscience
• Clinical Informatics

Background:

• Timely thrombolytic therapy is crucial for acute ischemic stroke outcomes.
• Manual review of clinical notes for contraindications is time-consuming and error-prone.
• Developing automated tools can enhance efficiency and accuracy.

Purpose of the Study:

• To develop and test a large language model (LLM)-based tool for identifying thrombolysis contraindications.
• To evaluate the performance of the LLM tool using synthetic clinical data.
• To assess the potential of LLMs in streamlining stroke treatment decision-making.

Main Methods:

• Generated 150 synthetic clinical notes with contraindications using LLMs.
• Employed Llama 3.1 405B with a custom prompt to extract contraindications.
• Evaluated performance using sensitivity, specificity, PPV, NPV, accuracy, and F1 score.

Main Results:

• The LLM tool achieved high performance metrics: 90.9% sensitivity, 99.2% specificity, 87.7% PPV, 99.4% NPV, 98.7% accuracy, and 0.892 F1 score.
• False positives were primarily due to irrelevant or repetitive information, with no observed hallucinations.
• The tool demonstrated robust identification of contraindications in synthetic data.

Conclusions:

• The LLM-based tool shows promise for accurately identifying stroke thrombolysis contraindications.
• Further validation with real-world EMR data is necessary.
• Integration into clinical workflows could facilitate faster and safer thrombolysis decisions.