Related Experiment Video
Updated: Nov 4, 2025
14:45
Transgenic Rodent Assay for Quantifying Male Germ Cell Mutant Frequency
Published on: August 6, 2014
16.5K
Permutation tests are robust and powerful at 0.5% and 5% significance levels
Kimihiro Noguchi1, Frank Konietschke2,3, Fernando Marmolejo-Ramos4
1Department of Mathematics, Western Washington University, Bellingham, WA, 98225, USA. Kimihiro.Noguchi@wwu.edu.
Behavior Research Methods
|May 29, 2021
Summary
To combat the replication crisis, researchers suggest a stricter significance level (α = 0.005). This study found permutation tests, like Welch t-test, are robust and powerful, unlike t-distribution tests under certain conditions.
Area of Science:
- Statistical methodology
- Psychological research methods
Background:
- The scientific community faces a replication crisis, increasing false positive findings.
- Suggestions include lowering the significance level (α) to 0.005 from 0.05.
Purpose of the Study:
- To evaluate the robustness and power of common statistical tests at α = 0.005 and α = 0.05.
- Investigate performance for metric and ordinal data in independent two-sample tests.
Main Methods:
- Extensive simulation study.
- Comparison of permutation tests (Welch t-test, Brunner-Munzel test) against t-distribution based tests.
- Analysis under skewed distributions with variance heterogeneity.
Main Results:
- Permutation versions of Welch t-test and Brunner-Munzel test demonstrated robustness and power at α = 0.005.
- Common t-distribution tests showed liberal or conservative behavior.
- T-distribution tests exhibited unusual power curve behavior with skewed data and unequal variances.
Conclusions:
- Permutation tests are recommended for their reliability at stricter significance levels.
- Caution is advised when using standard t-distribution tests under non-ideal data conditions.
Related Concept Videos
01:10Bonferroni Test
2.9K
The Bonferroni test is a statistical test named after Carlo Emilio Bonferroni, an Italian mathematician best known for Bonferroni inequalities. This statistical test is a type of multiple comparison test to determine which means are different than the rest. Bonferroni test can minimize the Type 1 error by reducing the significance level alpha, which otherwise increases with sample pairs.
The means of different samples are first paired in all possible combinations.
The null hypothesis of the...
The means of different samples are first paired in all possible combinations.
The null hypothesis of the...
2.9K
01:04Significance Testing: Overview
8.2K
Significance testing is a set of statistical methods used to test whether a claim about a parameter is valid. In analytical chemistry, significance testing is used primarily to determine whether the difference between two values comes from determinate or random errors. The effect of a particular change in the measurement protocol, analyst, or sample itself can cause a deviation from the expected result. In the case of a suspected deviation/outlier, we need to be able to confirm mathematically...
8.2K
01:14Errors In Hypothesis Tests
5.1K
When performing a hypothesis test, there are four possible outcomes depending on the actual truth (or falseness) of the null hypothesis and the decision to reject or not.
5.1K
01:17Sign Test for Matched Pairs
255
The sign test for matched pairs offers a robust method for comparing two paired samples, often for the effects of an intervention in one of them. This method is very useful in situations where the underlying distribution of the data is unknown. The test compares two related samples—often pre- and post-treatment measurements on the same subjects—to determine if there are significant differences in their median values.
To conduct the sign test, we first calculate the differences in...
To conduct the sign test, we first calculate the differences in...
255
01:17Wald-Wolfowitz Runs Test II
372
The Wald-Wolfowitz runs test, commonly referred to as the runs test, is a nonparametric test used to assess the randomness of ordered data. The test evaluates the number of runs, which are consecutive sequences of similar elements within the data. If the number of runs is significantly higher or lower than expected, the data is considered non-random, indicating a detectable pattern or structure.
For binary data, runs are identified using symbols such as + and −, or equivalently, 1s and 0s. In...
For binary data, runs are identified using symbols such as + and −, or equivalently, 1s and 0s. In...
372
01:24Testing a Claim about Population Proportion
3.5K
A complete procedure for testing a claim about a population proportion is provided here.
There are two methods of testing a claim about a population proportion: (1) Using the sample proportion from the data where a binomial distribution is approximated to the normal distribution and (2) Using the binomial probabilities calculated from the data.
The first method uses normal distribution as an approximation to the binomial distribution. The requirements are as follows: sample size is large...
There are two methods of testing a claim about a population proportion: (1) Using the sample proportion from the data where a binomial distribution is approximated to the normal distribution and (2) Using the binomial probabilities calculated from the data.
The first method uses normal distribution as an approximation to the binomial distribution. The requirements are as follows: sample size is large...
3.5K