Assessing Inaccuracies in Automated Information Extraction of Breast Imaging Findings

We previously identified breast imaging findings from radiology reports using an expert-based information extraction algorithm as part of the National Cancer Institute's Population-based Research Optimizing Screening through Personalized Regimens (PROSPR) initiative. We validate this algorithm and assess inaccuracies in a different institutional setting. Mammography, ultrasound (US), and breast magnetic resonance imaging (MRI) reports of patients at an academic health system between 4/2013 and 6/2013 were included for analysis. Accuracy of automatically extracting imaging findings using an algorithm developed at a different institution compared to manual gold standard review is reported. Extraction errors are further categorized based on manual review. Precision and recall for extracting BI-RADS categories remain between 0.9 and 1.0, except for MRI (0.7). F measures for extracting other findings are 0.9 for non-mass enhancement (in MRI) and 0.8-0.9 for cysts (in MRI and US). Extracting breast imaging findings resulted in lowest accuracy for findings of calcification (range 0.4-0.6 in mammography) and asymmetric density (0.5-0.7 in mammography). Majority of errors for extracting imaging findings were due to qualifier-based errors, descriptors which indicate absence of findings, missed by automated extraction (e.g., "benign" calcifications). Our information extraction algorithm provides an effective approach to extracting some breast imaging findings for populating a breast screening registry. However, errors in information extraction when utilizing methods in new settings demonstrate that further work is necessary to extract information content from unstructured multi-institutional radiology reports.