How Good Is Your Evidence and How Would You Know?

This paper examines the basic question of how we can come to form accurate beliefs about the world when we do not fully know how good or bad our evidence is. Here, we show, using simulations with otherwise optimal agents, the cost of misjudging the quality of our evidence. We compare different strategies for correctly estimating that quality, such as outcome- and expectation-based updating. We also identify conditions under which misjudgment of evidence quality can nevertheless lead to accurate beliefs, as well as those conditions where no strategy will help. These results indicate both where people will nevertheless succeed and where they will fail when information quality is degraded.