Comparison of electronic versus manual abstraction for 2 standardized perinatal care measures

Given that electronic clinical quality measures (eCQMs) are playing a central role in quality improvement applications nationwide, a stronger evidence base demonstrating their reliability is critically needed. To assess the reliability of electronic health record-extracted data elements and measure results for the Elective Delivery and Exclusive Breast Milk Feeding measures (vs manual abstraction) among a national sample of US acute care hospitals, as well as common sources of discrepancies and change over time.

eCQM and chart-abstracted data for the same patients were matched and compared at the data element and measure level for hospitals submitting both sources of data to The Joint Commission between 2017 and 2019. Sensitivity, specificity, and kappa statistics were used to assess reliability.

Although eCQM denominator reliability had moderate to substantial agreement for both measures and both improved over time (Elective Delivery: kappa = 0.59 [95% confidence interval (CI), 0.58-0.61] in 2017 and 0.84 [95% CI, 083-0.85] in 2019; Exclusive Breast Milk Feeding: kappa = 0.58 [95% CI, 0.54-0.62] in 2017 and 0.70 [95% CI, 0.67-0.73] in 2019), the numerator status reliability was poor for Elective Delivery (kappa = 0.08 [95% CI, 0.03-0.12] in 2017 and 0.10 [95% CI, 0.05-0.15] in 2019) but near perfect for Exclusive Breast Milk Feeding (kappa = 0.85 [0.83, 0.87] in 2017 and 0.84 [0.83, 0.85] in 2019). The failure of the eCQM to accurately capture estimated gestational age, conditions possibly justifying elective delivery, active labor, and medical induction were the main reasons for the discrepancies.

Although eCQM denominator reliability for the Elective Delivery and Exclusive Breast Milk Feeding measures had moderate agreement when compared to medical record review, the numerator status reliability was poor for Elective Delivery, but near perfect for Exclusive Breast Milk Feeding. Improvements in eCQM data capture of some key data elements would greatly improve the reliability.