Convergent validity and interrater reliability of estimating the ABCD2 score from medical records

The ABCD(2) score is increasingly used for risk stratification of transient ischemic attack patients. We sought to determine the reliability and convergent validity of retrospective ABCD(2) score estimation from medical records.

We compared ABCD(2) scores that were prospectively determined by a vascular neurology attending to scores determined retrospectively from medical record review. Emergency department records and neurology consult notes for patients with acute transient ischemic attack were abstracted with explicit ABCD(2) scoring redacted. Scores were estimated by 2 independent raters using these records. Estimated ABCD(2) component scores, total scores, and risk category were compared both between retrospective raters and with prospectively obtained scores. Reliability was assessed using unweighted κ statistics.

Interrater reliability was substantial with 72% exact agreement in total score between retrospective raters (κ=0.64) and nearly perfect with 82% agreement for ABCD(2) category (κ=0.71). Interrater agreement was best for age and diabetes mellitus and poorest for clinical features and duration. Agreement between the retrospective raters and prospectively obtained score was >90% for age, blood pressure, and diabetes mellitus, but only ≈70% for clinical features and duration. Retrospectively, estimated total ABCD(2) score exactly matched the prospective score in 58% of patients for rater 1 and 44% of patients for rater 2. Retrospectively, estimated ABCD(2) category matched the prospectively scored category in 67% of patients for rater 1 and 71% of patients for rater 2.

The ABCD(2) score can be abstracted from medical records with substantial interrater reliability but limited convergent validity. This may lead to misclassification of risk category in more than one third of patients.