Replication and p Intervals: p Values Predict the Future Only Vaguely, but Confidence Intervals Do Much Better

Area of Science:

• Psychological science
• Statistical methodology

Background:

• Replication is crucial for scientific validity.
• P-values are widely used in psychological research for statistical analysis.
• The utility of p-values in predicting replication is questionable.

Purpose of the Study:

• To evaluate the informativeness of p-values regarding scientific replication.
• To demonstrate the unreliability of p-values across repeated experiments.
• To advocate for alternative statistical methods that better support replicability.

Main Methods:

• A simulation study was conducted involving 25 repetitions of a typical experiment.
• Analysis focused on the variability of p-values in replicated studies.
• Calculations were performed to determine the probability distribution of p-values in replication attempts.

Main Results:

• P-values demonstrated significant variability, ranging from <.001 to .76 in simulations.
• An initial p-value of .05 yields an 80% probability for replication p-values within the wide interval (.00008, .44).
• This wide p-value interval persists regardless of sample size, indicating inherent unreliability.

Conclusions:

• P-values offer vague and unreliable information, making them inadequate for scientific inference.
• Confidence intervals provide superior insights into the replicability of findings.
• Researchers should reduce reliance on p-values, favoring confidence intervals, model-fitting, and meta-analytic approaches.