Physician inter-annotator agreement in the Quality Oncology Practice Initiative manual abstraction task

The Quality Oncology Practice Initiative (QOPI) relies on the accuracy of manual abstraction of clinical data from paper-based and electronic medical records (EMRs). Although there is no "gold standard" to measure manual abstraction accuracy, measurement of inter-annotator agreement (IAA) is a commonly agreed-on surrogate. We quantified the IAA of QOPI abstractions on a cohort of cancer patients treated at Beth Israel Deaconess Medical Center.

The EMR charts of 49 patients (20 colorectal cancer; 18 breast cancer; 11 non-Hodgkin lymphoma) were abstracted by separate physician abstractors in the fall 2010 and fall 2011 QOPI abstraction rounds. Cohen's kappa (κ) was calculated for encoded data; raw levels of agreement and magnitude of discrepancies were calculated for numeric and dated data.

One hundred two data elements with 2,035 paired entries were analyzed. Overall IAA for the 1,496 coded entries was κ = 0.75; median IAA for n = 85 individual coded elements was κ = 0.84 (interquartile range, 0.30 to 1.00). Overall IAA for the 421 dated entries was 73%; median IAA for n = 17 individual dated elements was 67% (interquartile range, 61% to 86%).

This study establishes a baseline level of IAA for a complex medical abstraction task with clear relevance for the oncology community. Given that the observed κ is considered only fair IAA, and that the rate of date discrepancy is high, caution is necessary in interpreting the results of QOPI and other manual abstractions of clinical oncology data. The accuracy of automated data extraction efforts, possibly including a future evolution of QOPI, will also need to be carefully evaluated.