Manually Abstracted versus Electronic Health Record Data for Surgical Quality Improvement

Manually abstracted variables are considered the gold standard within national surgical quality improvement (QI) programs. However, because of the resources associated with manual data abstraction, opportunities to automate data collection could have numerous benefits for surgical QI. The goal of this study is to describe the accuracy and concordance of Veterans Affairs (VA) Surgical Quality Improvement Program (VASQIP) electronic health record (EHR) variable correlates (derived using EHR data) when compared to manually abstracted VASQIP variables.

This was a national, cross-sectional analysis of VASQIP and VA Corporate Data Warehouse (i.e., EHR) data (2016-2020). EHR-derived VASQIP variable correlates were created from Corporate Data Warehouse and compared to manually abstracted VASQIP variables for the same cases. The primary measure of agreement was Cohen's kappa. Sensitivity, specificity, positive predictive value, and negative predictive value were also calculated for each variable with addition of exact match proportion for lab variables. Strong agreement was considered kappa ≥80%.

Among 533,164 cases across 113 hospitals for 429,163 unique patients, data were evaluated for five variable domains (race and ethnicity, preoperative risk factors, intraoperative factors, labs, and postoperative complications). Kappa for race and ethnicity ranged from 91.1 to 99.5%, with a median of 98.1% (IQR, 95.3-99.5%). Preoperative risk factors ranged from -0.1 to 83.0%, with a median of 28.6% (interquartile range [IQR], 12.7-53.9%). Preoperative labs ranged from 72.2 to 95.9% with a median of 91.9% (IQR, 89.9-93.3%). Intraoperative factors ranged from 0.0 to 99.5%, with a median of 93.9% (IQR, 9.9-97.3%). Postoperative complications ranged from 3.9 to 53.2%, with a median of 15.1% (IQR, 7.1-29.6%).

Apart from postoperative complications, data collection for many VASQIP variables could potentially be automated using EHR-derived correlates with a high level of accuracy. This could minimize the resources associated with manual data collection and increase the timeliness and robustness of surgical QI programs.