Bayesian Statistical Model of Item Response Theory in Observer Studies of Radiologists

The purpose of this study was to validate a Bayesian statistical model of item response theory (IRT). IRT was used to evaluate a new modality (temporal subtraction, TS) in observer studies of radiologists, compared with a conventional modality (computed tomography).

From previously published papers, we obtained two datasets of clinical observer studies of radiologists. Those studies used a multi-reader and multi-case paradigm to evaluate radiologists' detection abilities, primarily to determine if TS could enhance the detectability of bone metastasis or brain infarctions. We applied IRT to these studies' datasets using Stan software. Before applying IRT, the radiologists' responses were recorded as binaries for each case (1 = correct, 0 = incorrect). Effect of TS on detectability was evaluated by using our IRT model and calculating the 95% credible interval of the effect.

The mean, median, and 95% credible interval of the effect of TS were 0.913, 0.885, and 0.243-1.745 for the bone metastasis detection, and 2.524, 2.50, and 1.827-3.310, for the brain infarction detection. For both detection studies, the 95% credible intervals of the effect of TS did not include zero, indicating that TS significantly improved diagnostic ability.

Judgments based on the present study results were compatible with the two previous studies. Our study results demonstrated that the Bayesian statistical model of IRT could judge a new modality's usefulness.