Changes in methodological study characteristics in psychology between 2010-2021

Area of Science:

• Psychology
• Research Methodology
• Scientific Reproducibility

Background:

• The 2015 Open Science Collaboration replication project highlighted significant reproducibility issues in psychological science.
• Criticisms include reliance on null-hypothesis significance testing, small sample sizes, and questionable research practices.

Purpose of the Study:

• To analyze changes in empirical research practices in psychology over 12 years (2010-2015 and post-2015).
• To identify journal- and time-specific trends in statistical reporting and methodology.

Main Methods:

• Analysis of 57,909 articles from 12 journals using JATSdecoder software.
• Examination of trends in p-values, confidence intervals, directed tests, Bayesian inference, sample sizes, and multiple testing corrections.

Main Results:

• Median reported p-values per article decreased from 14 to 12; significant p-value proportion remained 69%.
• Reporting of effect sizes and confidence intervals increased; standard alpha level of .05 persisted.
• Use of multiple testing corrections decreased; Bayesian inference increased post-2015; median sample size rose from 105 to 190.

Conclusions:

• Despite some methodological shifts, fundamental issues impacting reproducibility in psychology persist.
• Increased sample sizes and a rise in Bayesian methods are noted, but p-value reporting and corrections for multiple testing show concerning trends.