Related Experiment Video
Updated: Jun 21, 2026
05:53
Candidate Gene Testing in Clinical Cohort Studies with Multiplexed Genotyping and Mass Spectrometry
Published on: June 21, 2018
What is the probability of replicating a statistically significant effect?
1University of Otago, Dunedin, New Zealand. miller@psy.otago.ac.nz
Psychonomic Bulletin & Review
|August 4, 2009
Summary
Determining the probability of replicating a statistically significant experimental effect is complex. Researchers often cannot precisely know this replication probability due to estimation limitations and unknown research context factors.
Area of Science:
- Statistics
- Research Methodology
- Scientific Replication
Background:
- Initial experiments often yield statistically significant results, raising questions about their reproducibility.
- The probability of replicating a significant finding is a fundamental concern in scientific research.
Purpose of the Study:
- To explore the interpretability and knowability of replication probabilities for statistically significant effects.
- To assess the practical limitations in estimating and understanding the likelihood of scientific replication.
Main Methods:
- Conceptual analysis of replication probability interpretations.
- Examination of statistical estimation limitations for initial experimental data.
- Discussion of contextual factors influencing reproducibility.
Main Results:
- The question of replication probability has two distinct interpretations.
- Estimating one type of replication probability from initial data is often imprecise and lacks practical utility.
- The second type of replication probability is inherently unknowable due to reliance on unknown research context.
Conclusions:
- Researchers generally cannot determine the probability of replicating a significant effect.
- Acceptance of this unknowability is necessary, regardless of the specific interpretation of replication probability sought.
- The practical utility of replication probability estimates in scientific research is severely limited.
Related Concept Videos
01:37Statistical Significance
Once data is collected from both the experimental and the control groups, a statistical analysis is conducted to find out if there are meaningful differences between the two groups. A statistical analysis determines how likely any difference found is due to chance (and thus not meaningful). In psychology, group differences are considered meaningful, or significant, if the odds that these differences occurred by chance alone are 5 percent or less. Stated another way, if we repeated this...
01:13Accuracy and Errors in Hypothesis Testing
Hypothesis testing is a fundamental statistical tool that begins with the assumption that the null hypothesis H0 is true. During this process, two types of errors can occur: Type I and Type II. A Type I error refers to the incorrect rejection of a true null hypothesis, while a Type II error involves the failure to reject a false null hypothesis.
In hypothesis testing, the probability of making a Type I error, denoted as α, is commonly set at 0.05. This significance level indicates a 5% chance...
In hypothesis testing, the probability of making a Type I error, denoted as α, is commonly set at 0.05. This significance level indicates a 5% chance...
01:16Statistical Hypothesis Testing
Hypothesis testing is a critical statistical procedure facilitating informed, evidence-based decisions. It begins with a hypothesis, which is a tentative explanation, or a prediction about a population parameter. This hypothesis can be either a null hypothesis (H0), indicating no effect or difference, or an alternative hypothesis (Ha), suggesting an effect or difference.
Statistical significance measures the probability that an observed result occurred by chance. If this probability, known as...
Statistical significance measures the probability that an observed result occurred by chance. If this probability, known as...
01:24Testing a Claim about Population Proportion
A complete procedure for testing a claim about a population proportion is provided here.
There are two methods of testing a claim about a population proportion: (1) Using the sample proportion from the data where a binomial distribution is approximated to the normal distribution and (2) Using the binomial probabilities calculated from the data.
The first method uses normal distribution as an approximation to the binomial distribution. The requirements are as follows: sample size is large...
There are two methods of testing a claim about a population proportion: (1) Using the sample proportion from the data where a binomial distribution is approximated to the normal distribution and (2) Using the binomial probabilities calculated from the data.
The first method uses normal distribution as an approximation to the binomial distribution. The requirements are as follows: sample size is large...
01:04Significance Testing: Overview
Significance testing is a set of statistical methods used to test whether a claim about a parameter is valid. In analytical chemistry, significance testing is used primarily to determine whether the difference between two values comes from determinate or random errors. The effect of a particular change in the measurement protocol, analyst, or sample itself can cause a deviation from the expected result. In the case of a suspected deviation/outlier, we need to be able to confirm mathematically...
01:14Errors In Hypothesis Tests
When performing a hypothesis test, there are four possible outcomes depending on the actual truth (or falseness) of the null hypothesis and the decision to reject or not.