Accuracy of clinicians and models for estimating the probability that a pulmonary nodule is malignant

Management of pulmonary nodules depends critically on the probability of malignancy. Models to estimate probability have been developed and validated, but most clinicians rely on judgment.

The aim of this study was to compare the accuracy of clinical judgment with that of two prediction models.

Physician participants reviewed up to five clinical vignettes, selected at random from a larger pool of 35 vignettes, all based on actual patients with lung nodules of known final diagnosis. Vignettes included clinical information and a representative slice from computed tomography. Clinicians estimated the probability of malignancy for each vignette. To examine agreement with models, we calculated intraclass correlation coefficients (ICC) and kappa statistics. To examine accuracy, we compared areas under the receiver operator characteristic curve (AUC).

Thirty-six participants completed 179 vignettes, 47% of which described patients with malignant nodules. Agreement between participants and models was fair for the Mayo Clinic model (ICC, 0.37; 95% confidence interval [CI], 0.23-0.50) and moderate for the Veterans Affairs model (ICC, 0.46; 95% CI, 0.34-0.57). There was no difference in accuracy between participants (AUC, 0.70; 95% CI, 0.62-0.77) and the Mayo Clinic model (AUC, 0.71; 95% CI, 0.62-0.80; P = 0.90) or the Veterans Affairs model (AUC, 0.72; 95% CI, 0.64-0.80; P = 0.54).

In this vignette-based study, clinical judgment and models appeared to have similar accuracy for lung nodule characterization, but agreement between judgment and the models was modest, suggesting that qualitative and quantitative approaches may provide complementary information.