在研究中积累证据:一致的方法可以防止p-hacking产生的错误发现.
Duane T Wegener1, Jolynn Pek1, Leandre R Fabrigar2
1Department of Psychology, Ohio State University, Columbus, Ohio, United States of America.
PloS one
|August 29, 2024
概括
灵活的分析策略,称为p-hacking,可以导致单个研究中的假阳性结果. 然而,在多项研究中使用一致的方法可以显著降低p-hacking产生可靠错误结果的风险.
科学领域:
- 方法论 方法论 方法论
- 统计推理 统计推理
- 经验研究 实证研究
背景情况:
- 经验科学经常评估对观察到的数据的相互竞争的解释.
- 在统计测试中",显著"的p值通常表明无效 (零) 效应的不合理性.
- 对于p-hacking或灵活的分析策略存在担忧,可能会导致虚假阳性率膨胀.
研究的目的:
- 调查p-hacking对多项研究发现的可靠性的影响.
- 评估跨研究一致的方法是否能减轻p-hacking的影响.
- 评估p-hacking作为对一致实证结果的解释的可信性.
主要方法:
- 对研究组进行模拟,以模拟p-hacking的影响.
- 对单个研究的错误发现率与使用一致方法的研究集进行比较.
- 检查了选择性报道和不同程度的p-hacking的影响.
主要成果:
- 跨研究的一致方法大大降低了p-hacking产生重大结果的可能性.
- 黑客攻击需要大量的选择性报告和严重的故意操纵,以在研究中产生一致的错误结果.
- 在单一的大样本研究中,P-hacking可以产生高的假阳性率,但在方法上一致的研究系列中效果较差.
结论:
- 方法上一致的研究集对p-hacking有很强的抵抗力,提高了经验研究结果的可靠性.
- 通过统一的方法在多项研究中得到一致的结果,P-hacking是不太可能的解释.
- 使用一致方法的系列研究提供了比单个大样本研究更好的防范假阳性.
相关概念视频
Decision Making: P-value Method
5.3K
The process of hypothesis testing based on the P-value method includes calculating the P- value using the sample data and interpreting it.
First, a specific claim about the population parameter is proposed. The claim is based on the research question and is stated in a simple form. Further, an opposing statement to the claim is also stated. These statements can act as null and alternative hypotheses: a null hypothesis would be a neutral statement while the alternative hypothesis can...
First, a specific claim about the population parameter is proposed. The claim is based on the research question and is stated in a simple form. Further, an opposing statement to the claim is also stated. These statements can act as null and alternative hypotheses: a null hypothesis would be a neutral statement while the alternative hypothesis can...
5.3K
Accuracy and Errors in Hypothesis Testing
183
Hypothesis testing is a fundamental statistical tool that begins with the assumption that the null hypothesis H0 is true. During this process, two types of errors can occur: Type I and Type II. A Type I error refers to the incorrect rejection of a true null hypothesis, while a Type II error involves the failure to reject a false null hypothesis.
In hypothesis testing, the probability of making a Type I error, denoted as α, is commonly set at 0.05. This significance level indicates a 5%...
In hypothesis testing, the probability of making a Type I error, denoted as α, is commonly set at 0.05. This significance level indicates a 5%...
183
Decision Making: Traditional Method
4.0K
The process of hypothesis testing based on the traditional method includes calculating the critical value, testing the value of the test statistic using the sample data, and interpreting these values.
First, a specific claim about the population parameter is decided based on the research question and is stated in a simple form. Further, an opposing statement to this claim is also stated. These statements can act as null and alternative hypotheses, out of which a null hypothesis would be a...
First, a specific claim about the population parameter is decided based on the research question and is stated in a simple form. Further, an opposing statement to this claim is also stated. These statements can act as null and alternative hypotheses, out of which a null hypothesis would be a...
4.0K
Statistical Significance
20.1K
Once data is collected from both the experimental and the control groups, a statistical analysis is conducted to find out if there are meaningful differences between the two groups. A statistical analysis determines how likely any difference found is due to chance (and thus not meaningful). In psychology, group differences are considered meaningful, or significant, if the odds that these differences occurred by chance alone are 5 percent or less. Stated another way, if we repeated this...
20.1K
Bonferroni Test
2.7K
The Bonferroni test is a statistical test named after Carlo Emilio Bonferroni, an Italian mathematician best known for Bonferroni inequalities. This statistical test is a type of multiple comparison test to determine which means are different than the rest. Bonferroni test can minimize the Type 1 error by reducing the significance level alpha, which otherwise increases with sample pairs.
The means of different samples are first paired in all possible combinations.
The null hypothesis of the...
The means of different samples are first paired in all possible combinations.
The null hypothesis of the...
2.7K
Bias
3.9K
Bias refers to any tendency that prevents a question from being considered unprejudiced. In research, bias occurs when one outcome or answer is selected or encouraged over others in sampling or testing. Bias can occur during any research phase, including study design, data collection, analysis, and publication.
In statistics, a sampling bias is created when a sample is collected from a population, and some members of the population are not as likely to be chosen as others (remember, each member...
In statistics, a sampling bias is created when a sample is collected from a population, and some members of the population are not as likely to be chosen as others (remember, each member...
3.9K


