Reevaluating excess success in psychological science

Area of Science:

• Psychological Science
• Research Methodology
• Scientific Integrity

Background:

• A previous analysis claimed 82% of articles in Psychological Science (2009-2012) with >=4 experiments were untrustworthy.
• This claim has raised concerns about the reliability of published psychological research.

Purpose of the Study:

• To critique Francis's analysis of research reliability in Psychological Science.
• To re-evaluate the trustworthiness of articles flagged by Francis, focusing on methodological choices.
• To assess the evidential value of studies using p-curve analysis.

Main Methods:

• Critiqued Francis's statistical approach, highlighting sensitivity to experiment and significance test inclusion.
• Re-analyzed a specific article (van Boxtel & Koch, 2012) flagged by Francis.
• Applied p-curve analysis to 44 articles examined in Francis's study.

Main Results:

• Francis's analysis may have mistakenly flagged articles due to inappropriate inclusion of experiments and tests.
• Decisions on including specific tests potentially impacted 34 of 44 analyzed articles.
• P-curve analysis revealed 9 studies with significant evidential value, 11 with insufficient value, and 1 with p-hacking evidence.

Conclusions:

• Francis's conclusion that many Psychological Science articles are untrustworthy is likely insufficient and potentially misleading.
• The specific case of van Boxtel & Koch (2012) demonstrates flaws in Francis's methodology.
• A nuanced re-evaluation using p-curves suggests varying levels of evidential value, not wholesale untrustworthiness.