Bigram frequency analysis for detection of radiology report errors

This pilot study evaluates the utility of analyzing bigram frequencies for detecting radiology report errors.

A corpus of 48,050 CT reports was used to enumerate the frequency of each bigram (F_AB), and the expected frequency of each bigram in the corpus based on the constituent unigram frequencies (P_AB). A test set consisted of a separate random sample of 200 radiology reports dictated by attendings for CT scans of the abdomen in 2019, as well as a random sample of 200 radiology reports for CT scans of the abdomen dictated in 2019 by 52 different residents or fellows prior to editing by the signing attendings. Bigrams in the test reports that occurred either rarely or not at all in the corpus were flagged for manual review by an abdominal radiologist.

Of 682 n-grams flagged in attending reports, 11.6% were true errors, while of 1378 n-grams flagged in trainee reports, 7.9% were true errors. The largest group of flagged n-grams in both test sets involved bigrams that did not appear in the corpus, but whose constituent words did appear in the corpus. Subsets of 50 attending and 50 resident reports were manually reviewed, revealing that the flagging procedure had a sensitivity for errors of 58% (22/38) in the attending reports and 97% (31/32) in the resident reports.

Bigram frequency analysis may be of practical value in reviewing radiology reports for errors. Further methodological refinement to improve the positive predictive value of error detection is required.