Crowdsourcing hypothesis tests: Making transparent how design choices shape research results

Area of Science:

• Psychology
• Social Psychology
• Cognitive Science

Background:

• Subjective decisions in study design can influence research outcomes.
• Understanding the extent of this influence is crucial for scientific reproducibility.

Purpose of the Study:

• To investigate how scientists' subjective decisions in study design affect research results.
• To assess the variability in effect sizes and statistical significance across different study designs for the same research questions.

Main Methods:

• Fifteen research teams independently designed studies for five research questions.
• Large, diverse participant samples (N > 15,000) were used, with random assignment to study versions.
• Meta-analysis and Bayesian methods were employed to analyze effect size variability.

Main Results:

• Effect sizes varied dramatically, with significant effects in opposite directions for 4 out of 5 hypotheses.
• Variability in results was primarily attributed to the hypothesis tested, not the research team's skill.
• Scientists' predictions correlated with study outcomes, suggesting predictable biases.

Conclusions:

• Crowdsourced testing of hypotheses can reveal the true consistency of empirical support.
• Subjective design choices introduce considerable variability, impacting the reliability of single studies.
• The findings underscore the importance of transparent and collaborative research practices.