Communicating uncertain experimental evidence

Four experiments examined when laypeople attribute unexpected experimental outcomes to error, in foresight and in hindsight, along with their judgments of whether the data should be published. Participants read vignettes describing hypothetical experiments, along with the result of the initial observation, considered as either a possibility (foresight) or a reality (hindsight). Experiment 1 found that the initial observation seemed more likely to be replicated when viewed in hindsight than in foresight. The remaining experiments contrasted responses to an initial observation from 1 of the 4 studies that was either expected or unexpected (based on the predictions of participants in Experiment 1). Experiments 2A-C and Experiment 3 found that unexpected results were more likely to be attributed to methodological problems than were expected ones-but to the same degree in foresight and in hindsight. Participants in Experiment 4 had more confidence in an explanation for an unexpected outcome when it was mentioned before that outcome was revealed than when it was suggested only after the surprise was known. In all the experiments, most participants recommended collecting more data before publishing the results, especially when they attributed the results to multiple causes. The results suggest that considering the causes of unexpected experimental results in foresight may improve the evaluation and communication of those results in hindsight.