Accuracy of Large Language Models to Identify Stroke Subtypes Within Unstructured Electronic Health Record Data

Area of Science:

• Medical Informatics
• Artificial Intelligence in Medicine
• Clinical Data Analysis

Background:

• Accurate stroke classification from electronic health records is difficult due to limited structured data.
• Manual review of clinical documentation is often required for precise stroke typing.
• This study investigates the utility of large language models for automated stroke classification.

Purpose of the Study:

• To evaluate the accuracy of GPT-4o in classifying stroke types (ischemic vs. hemorrhagic) and ischemic stroke subtypes.
• To assess the performance of different prompting strategies for GPT-4o in this task.
• To compare AI-driven classification with expert manual abstraction.

Main Methods:

• A retrieval-augmented generation framework using GPT-4o was developed for stroke classification.
• Data from the American Heart Association Get With The Guidelines-Stroke registry was used as the gold standard.
• Three prompting strategies (zero-shot Chain-of-Thought, expert-guided, instruction-based) were tested on EHR data from two health systems.

Main Results:

• GPT-4o achieved 98% accuracy in classifying stroke type (ischemic vs. hemorrhagic) in external validation.
• Sensitivity and specificity for stroke type classification were high (0.98 and 0.97, respectively).
• Performance varied for ischemic stroke subtypes, with high accuracy for cardioembolism (0.98 specificity) but lower for cryptogenic stroke (0.40 sensitivity).

Conclusions:

• GPT-4o shows high accuracy for differentiating between ischemic and hemorrhagic stroke types.
• The model faces limitations in accurately classifying various ischemic stroke subtypes.
• Zero-shot Chain-of-Thought prompting proved effective, requiring minimal human input.