Predicting Findings from Radiology Indication Text: BERT vs Naïve Bayes Classification

This retrospective study evaluated Naïve Bayes (NB) and transformer-based BERT models for predicting findings from radiology order Indication text for use in radiology informatics and automated worklist triage systems. From the MIMIC-CXR-JPG dataset, 219,995 chest radiograph orders were extracted, and the indication text was paired with CheXpert-generated labels for a binary (findings/none) and multi-label (13-category) classification task. Three BERT models (baseBERT, radBERT, BioClinicalBERT) were compared to NB classifiers using bootstrap confidence intervals, McNemar's test, and paired permutation tests. Precision-recall curves assessed triage utility under a high-precision constraint. On the binary task, all BERT models significantly outperformed NB on AUROC (0.793-0.798 vs. 0.763; $P<0.001$ ). The radBERT model did better on AUROC than baseBERT and BioClinicalBERT (0.798 vs. 0.793 for both; $P<0.001$ , DeLong test) by a small but statistically significant amount. On the multi-label task, NB had better micro-F1 at the default threshold (0.399 vs. 0.361-0.388; $P<0.001$ ), but the BERT models had better AUPRC (0.470-0.475 vs. 0.358) and calibration (expected calibration error 0.011-0.016 vs. 0.077). NB's apparent advantage was due to its overconfident probabilities, not overall superiority. At a 90% precision constraint, radBERT recalled 55.0% of positive orders (36,557 examinations triaged per 100,000 annual orders). At 95% precision, radBERT maintained an actionable recall of 26.4%, whereas NB's recall collapsed to near zero. The modest edge of radBERT over general-domain and clinical-note-pretrained variants on the binary task suggests the distinctive character of Indication text limits the benefit of domain-specific pretraining. Order Indication text contains actionable information for predicting imaging findings, supporting transformer-based models for automated radiography triage systems.