Derivation, Validation, Sustained Performance, and Clinical Impact of an Electronic Medical Record-Based...

Postoperative delirium is an important problem for surgical inpatients and was the target of a multidisciplinary quality improvement project at our institution. We developed and tested a semiautomated delirium risk stratification instrument, Age, WORLD backwards, Orientation, iLlness severity, Surgery-specific risk (AWOL-S), in 3 independent cohorts from our tertiary care hospital and describe its performance characteristics and impact on clinical care.

The risk stratification instrument was derived with elective surgical patients who were admitted at least overnight and received at least 1 postoperative delirium screen (Nursing Delirium Screening Scale [NuDESC] or Confusion Assessment Method for the Intensive Care Unit [CAM-ICU]) and preoperative cognitive screening tests (orientation to place and ability to spell WORLD backward). Using data pragmatically collected between December 7, 2016, and June 15, 2017, we derived a logistic regression model predicting probability of delirium in the first 7 postoperative hospital days. A priori predictors included age, cognitive screening, illness severity or American Society of Anesthesiologists physical status, and surgical delirium risk. We applied model odds ratios to 2 subsequent cohorts ("validation" and "sustained performance") and assessed performance using area under the receiver operator characteristic curves (AUC-ROC). A post hoc sensitivity analysis assessed performance in emergency and preadmitted patients. Finally, we retrospectively evaluated the use of benzodiazepines and anticholinergic medications in patients who screened at high risk for delirium.

The logistic regression model used to derive odds ratios for the risk prediction tool included 2091 patients. Model AUC-ROC was 0.71 (0.67-0.75), compared with 0.65 (0.58-0.72) in the validation (n = 908) and 0.75 (0.71-0.78) in the sustained performance (n = 3168) cohorts. Sensitivity was approximately 75% in the derivation and sustained performance cohorts; specificity was approximately 59%. The AUC-ROC for emergency and preadmitted patients was 0.71 (0.67-0.75; n = 1301). After AWOL-S was implemented clinically, patients at high risk for delirium (n = 3630) had 21% (3%-36%) lower relative risk of receiving an anticholinergic medication perioperatively after controlling for secular trends.

The AWOL-S delirium risk stratification tool has moderate accuracy for delirium prediction in a cohort of elective surgical patients, and performance is largely unchanged in emergent/preadmitted surgical patients. Using AWOL-S risk stratification as a part of a multidisciplinary delirium reduction intervention was associated with significantly lower rates of perioperative anticholinergic but not benzodiazepine, medications in those at high risk for delirium. AWOL-S offers a feasible starting point for electronic medical record-based postoperative delirium risk stratification and may serve as a useful paradigm for other institutions.