Assessment of the Modified Rankin Scale in Electronic Health Records With a Fine-Tuned Large Language Model:...

Area of Science:

• Neurology
• Artificial Intelligence
• Medical Informatics

Background:

• The modified Rankin scale (mRS) is crucial for stroke research outcomes.
• Retrospective mRS assessment from electronic health records (EHRs) is labor-intensive and variable.
• Large language models (LLMs) show potential for automating text classification tasks.

Purpose of the Study:

• To develop and evaluate a fine-tuned LLM for classifying mRS scores from EHR text.
• To assess LLM performance in analyzing stroke patient outcomes for clinical and research use.

Main Methods:

• Retrospective cohort study of 2290 patients from August 2020 to June 2023.
• Two independent researchers assigned mRS scores at discharge and 90 days post-discharge.
• Trained multiclass (7 scores) and binary (independence vs. non-independence) LLMs on EHR data, using cross-validation.

Main Results:

• The multiclass LLM achieved 77% accuracy and 0.92 Cohen κ.
• The binary LLM achieved 92% accuracy and 0.84 Cohen κ for functional independence.
• Highest accuracy for mRS score 4 (90%), lowest for mRS score 2 (28%).

Conclusions:

• LLMs can be successfully trained to determine mRS scores from EHR text.
• The developed LLM shows high accuracy, particularly for binary classification of functional independence.
• Further refinement is needed to improve discrimination between intermediate mRS scores.