When Cognitive Reflection Leads to Less Overall but More Systematic Judgment Bias: The Case of the Base Rates Fallacy

Although widely used in the judgment under uncertainty literature, the so-called Lawyer-Engineer problem does not have a Bayesian solution because the base rates typically oppose qualitative stereotypical information, which has an undefined diagnostic value. We propose an experimental paradigm that elicits participants' subjective estimates of the diagnosticity of stereotypical information and allows us to investigate the degree to which participants are able to integrate both sources of information (base rates and stereotypical descriptions) according to the Bayesian rule. This paradigm was used to test the hypothesis that the responses (probability estimates) to the Lawyer-Engineer problem from more rational individuals deviate from normative Bayesian solutions in a way that shows smaller but more systematic bias. The results further suggest that the estimates of less rational participants are noisier (less reliable) but may be more accurate when aggregated across several problems.