The impact of problem domain on Bayesian inferences: A systematic investigation

Sparse (and occasionally contradictory) evidence exists regarding the impact of domain on probabilistic updating, some of which suggests that Bayesian word problems with medical content may be especially challenging. The present research aims to address this gap in knowledge through three pre-registered online studies, which involved a total of 2,238 participants. Bayesian word problems were related to one of three domains: medical, daily-life, and abstract. In the first two cases, problems presented realistic content and plausible numerical information, while in the latter, problems contained explicitly imaginary elements. Problems across domains were matched in terms of all relevant statistical values and, as much as possible, wording. Studies 1 and 2 utilized the same set of problems, but different response elicitation methods (i.e., an open-ended and a multiple-choice question, respectively). Study 3 involved a larger number of participants per condition and a smaller set of problems to more thoroughly investigate the magnitude of differences between the domains. There was a generally low rate of correct responses (17.2%, 17.4%, and 14.3% in Studies 1, 2, and 3, respectively), consistent with accuracy levels commonly observed in the literature for this specific task with online samples. Nonetheless, a small but significant difference between domains was observed: participants' accuracy did not differ between medical and daily-life problems, while it was significantly higher in corresponding abstract problems. These results suggest that medical problems are not inherently more difficult to solve, but rather that performance is improved with abstract problems for which participants cannot draw from their background knowledge.