Limited Usefulness of Capture Procedure and Capture Percentage for Evaluating Reproducibility in Psychological...

Area of Science:

• Psychological science
• Research methodology

Background:

• Reproducibility is a growing concern in psychological science.
• Previous studies, such as Replication Project: Psychology, reported low replication rates (41%).
• The capture procedure (CPro) and capture percentage (CPer) are common metrics used to assess replication success.

Purpose of the Study:

• To evaluate the accuracy of the capture procedure (CPro) and capture percentage (CPer) in determining replication success.
• To identify potential biases in CPro and CPer when population effect sizes or sample sizes differ between original and replicated studies.

Main Methods:

• The study employed simulation methods to model replication scenarios.
• Various conditions were simulated, including differing population effect sizes and sample sizes between original and replicated studies.

Main Results:

• Simulation results indicate that CPro and CPer can be biased.
• These biases can lead to incorrect conclusions regarding the success or failure of replication.
• Specifically, incorrect conclusions may arise when population effect sizes differ or when sample sizes differ despite identical population effect sizes.

Conclusions:

• The findings suggest that CPro and CPer may not be reliable indicators of replication success under certain conditions.
• Researchers may need to exercise caution when interpreting replication results based on these metrics.
• Further research into more robust methods for assessing reproducibility in psychological science is warranted.