Assessment of the Modified Rankin Scale in Electronic Health Records with a Fine-tuned Large Language Model

Area of Science:

• Neurology
• Medical Informatics
• Artificial Intelligence

Background:

• The modified Rankin scale (mRS) is crucial for stroke research outcomes.
• Retrospective mRS assessment from electronic health records (EHR) is labor-intensive and variable.
• Large language models (LLMs) show promise in automating clinical text classification.

Purpose of the Study:

• To evaluate the efficacy of a fine-tuned LLM in analyzing EHR text for mRS score classification.
• To develop automated tools for mRS scoring in clinical and research settings.

Main Methods:

• Retrospective cohort study of stroke patients (August 2020 - June 2023).
• Two independent researchers assigned mRS scores at discharge and 90 days post-discharge.
• Trained multiclass (7 scores) and binary (independence vs. non-independence) LLM models on EHR data.

Main Results:

• Multiclass model achieved 77% accuracy and 0.92 Cohen's Kappa.
• Binary model achieved 92% accuracy and 0.84 Cohen's Kappa.
• Highest class-specific accuracy for mRS 4 (90%), lowest for mRS 2 (28%).

Conclusions:

• LLMs can be successfully trained to determine mRS scores from EHR text.
• Automated LLMs can enhance scalability for large clinical datasets.
• Potential for data-driven public health strategies and resource allocation in stroke care.