Assessment of the Modified Rankin Scale in Electronic Health Records with a Fine-tuned Large Language Model

Area of Science:

• Neurology
• Medical Informatics
• Artificial Intelligence

Background:

• The modified Rankin scale (mRS) is crucial for stroke research outcomes.
• Retrospective mRS assessment from electronic health records (EHR) is labor-intensive and variable.
• Large language models (LLMs) show promise for automating text classification tasks.

Purpose of the Study:

• To develop and evaluate a fine-tuned LLM for classifying mRS scores from EHR text.
• To enable efficient and reliable mRS scoring for clinical and research applications.

Main Methods:

• Retrospective cohort study of stroke patients (August 2020 - June 2023).
• Two independent researchers assigned mRS scores at discharge and 90 days post-discharge.
• Trained multiclass (7 scores) and binary (independent vs. non-independent) LLMs on EHR data.
• Evaluated models using accuracy and Cohen's kappa with four-fold cross-validation.

Main Results:

• 2,290 EHR passages with mRS scores were used for training.
• Multiclass model achieved 77% accuracy and 0.92 Cohen's kappa.
• Binary model achieved 92% accuracy and 0.84 Cohen's kappa.
• Highest accuracy for mRS 4 (90%), lowest for mRS 2 (28%).

Conclusions:

• LLMs can be successfully trained to determine mRS scores from EHR text.
• The developed LLM demonstrates high accuracy for binary classification of functional independence.
• Further improvements are needed to enhance discrimination between intermediate mRS scores.