Evaluating the role of mental sampling in probability judgments: Illogical rankings occur in a predictable manner

People's probability judgments often appear to be probabilistically incoherent, as exemplified by the conjunction fallacy. Recently, various sampling-based models have been proposed as an integrative account for different biases and fallacies in probability judgments. In the current study, the novel Event Ranking Task was used to investigate sampling-based models of probability judgments. On each trial of the Event Ranking Task, participants were asked to provide a ranking for an event set consisting of four events, A, not-A, B, and not-B, in terms of their perceived likelihoods. Qualitative predictions were formally derived by assuming direct sampling from a fixed underlying probability distribution. Adding read-out noise in the sampling process - as suggested in the Probability Theory plus Noise model (Costello and Watts, 2014) - did not change the qualitative predictions. Two online experiments, where participants ranked twelve different event sets, yielded results in line with the qualitative predictions, providing evidence for the idea that mental sampling underlies probability judgments.