Jove
Visualize
Contact Us
JoVE
x logofacebook logolinkedin logoyoutube logo
ABOUT JoVE
OverviewLeadershipBlogJoVE Help Center
AUTHORS
Publishing ProcessEditorial BoardScope & PoliciesPeer ReviewFAQSubmit
LIBRARIANS
TestimonialsSubscriptionsAccessResourcesLibrary Advisory BoardFAQ
RESEARCH
JoVE JournalMethods CollectionsJoVE Encyclopedia of ExperimentsArchive
EDUCATION
JoVE CoreJoVE BusinessJoVE Science EducationJoVE Lab ManualFaculty Resource CenterFaculty Site
Terms & Conditions of Use
Privacy Policy
Policies

Related Concept Videos

Significance Testing: Overview01:04

Significance Testing: Overview

9.1K
Significance testing is a set of statistical methods used to test whether a claim about a parameter is valid. In analytical chemistry, significance testing is used primarily to determine whether the difference between two values comes from determinate or random errors. The effect of a particular change in the measurement protocol, analyst, or sample itself can cause a deviation from the expected result. In the case of a suspected deviation/outlier, we need to be able to confirm mathematically...
9.1K
Statistical Hypothesis Testing01:16

Statistical Hypothesis Testing

4.8K
Hypothesis testing is a critical statistical procedure facilitating informed, evidence-based decisions. It begins with a hypothesis, which is a tentative explanation, or a prediction about a population parameter. This hypothesis can be either a null hypothesis (H0), indicating no effect or difference, or an alternative hypothesis (Ha), suggesting an effect or difference.
Statistical significance measures the probability that an observed result occurred by chance. If this probability, known as...
4.8K
Statistical Significance01:50

Statistical Significance

20.7K
Once data is collected from both the experimental and the control groups, a statistical analysis is conducted to find out if there are meaningful differences between the two groups. A statistical analysis determines how likely any difference found is due to chance (and thus not meaningful). In psychology, group differences are considered meaningful, or significant, if the odds that these differences occurred by chance alone are 5 percent or less. Stated another way, if we repeated this...
20.7K
Accuracy and Errors in Hypothesis Testing01:13

Accuracy and Errors in Hypothesis Testing

432
Hypothesis testing is a fundamental statistical tool that begins with the assumption that the null hypothesis H0 is true. During this process, two types of errors can occur: Type I and Type II. A Type I error refers to the incorrect rejection of a true null hypothesis, while a Type II error involves the failure to reject a false null hypothesis.
In hypothesis testing, the probability of making a Type I error, denoted as α, is commonly set at 0.05. This significance level indicates a 5%...
432
Identifying Statistically Significant Differences: The F-Test01:14

Identifying Statistically Significant Differences: The F-Test

2.8K
The F-test is used to compare two sample variances to each other or compare the sample variance to the population variance. It is used to decide whether an indeterminate error can explain the difference in their values. The underlying assumptions that allow the use of the F-test include the data set or sets are normally distributed, and the data sets are independent of each other. The test statistic F is calculated by dividing one variance by another. In other words, the square of one standard...
2.8K
Fisher's Exact Test01:08

Fisher's Exact Test

980
Fisher's exact test is a statistical significance test widely used to analyze 2x2 contingency tables, particularly in situations where sample sizes are small. Unlike the chi-squared test, which approximates P-values and assumes minimum expected frequencies of at least five in each cell, Fisher's exact test calculates the exact probability (P-value) of observing the data or more extreme results under the null hypothesis. This feature makes it especially valuable when the assumptions of...
980

You might also read

Related Articles

Articles linked to this work by shared authors, journal, and citation graph.

Sort by
Same author

Stage 2 Registered Report Personal factors and group creative outcomes: A correlational meta-analysis.

F1000Research·2025
Same author

The Influence of Posture on Attention.

Experimental psychology·2023
Same author

Left-truncated effects and overestimated meta-analytic means.

Proceedings of the National Academy of Sciences of the United States of America·2022
Same author

rmcorrShiny: A web and standalone application for repeated measures correlation.

F1000Research·2021
Same author

Corrigendum: Repeated Measures Correlation.

Frontiers in psychology·2019
Same author

Linking the Divergent and Convergent Processes of Collaborative Creativity: The Impact of Expertise Levels and Elaboration Processes.

Frontiers in psychology·2019

Related Experiment Video

Updated: Nov 22, 2025

Mapping Dysfunctional Protein-Protein Interactions in Disease
09:39

Mapping Dysfunctional Protein-Protein Interactions in Disease

Published on: October 24, 2025

256

Statistical Significance Filtering Overestimates Effects and Impedes Falsification: A Critique of.

Jonathan Z Bakdash1,2, Laura R Marusich3, Jared B Kenworthy4

  • 1United States Army Combat Capabilities Development Command, Army Research Laboratory South at the University of Texas at Dallas, Richardson, TX, United States.

Frontiers in Psychology
|January 8, 2021
PubMed
Summary

Selecting statistically significant results in meta-analysis inflates effect sizes. This study shows significance filtering overestimated correlations by 56%, hindering scientific falsification and established research practices.

Keywords:
confirmation biasfalsificationmeta-analysisp-hackingperformanceselection biassignificance filtersituation awareness

More Related Videos

Making Record-efficiency SnS Solar Cells by Thermal Evaporation and Atomic Layer Deposition
14:01

Making Record-efficiency SnS Solar Cells by Thermal Evaporation and Atomic Layer Deposition

Published on: May 22, 2015

43.1K
Improving Reproducibility to Meet Minimal Information for Studies of Extracellular Vesicles 2018 Guidelines in Nanoparticle Tracking Analysis
08:52

Improving Reproducibility to Meet Minimal Information for Studies of Extracellular Vesicles 2018 Guidelines in Nanoparticle Tracking Analysis

Published on: November 17, 2021

2.6K

Related Experiment Videos

Last Updated: Nov 22, 2025

Mapping Dysfunctional Protein-Protein Interactions in Disease
09:39

Mapping Dysfunctional Protein-Protein Interactions in Disease

Published on: October 24, 2025

256
Making Record-efficiency SnS Solar Cells by Thermal Evaporation and Atomic Layer Deposition
14:01

Making Record-efficiency SnS Solar Cells by Thermal Evaporation and Atomic Layer Deposition

Published on: May 22, 2015

43.1K
Improving Reproducibility to Meet Minimal Information for Studies of Extracellular Vesicles 2018 Guidelines in Nanoparticle Tracking Analysis
08:52

Improving Reproducibility to Meet Minimal Information for Studies of Extracellular Vesicles 2018 Guidelines in Nanoparticle Tracking Analysis

Published on: November 17, 2021

2.6K

Area of Science:

  • Psychology
  • Cognitive Science
  • Research Methodology

Background:

  • Selecting study results based on statistical significance can lead to overestimated effect sizes.
  • This practice impedes the falsification of scientific hypotheses.
  • Previous quantitative syntheses have employed significance-based selection methods.

Purpose of the Study:

  • To critique a quantitative synthesis that used statistical significance to select effects for situation awareness-performance associations.
  • To quantitatively evaluate the impact of significance-based filtering on meta-analytic results.
  • To compare effect sizes derived from significance-filtered data versus all reported effects.

Main Methods:

  • Critiqued a meta-analysis by Endsley (2019) that used significance to score and select effects.
  • Compared results from significance-filtered effects against analyses including all reported effects.
  • Evaluated the impact of filtering on mean correlations and effect size distributions.

Main Results:

  • Significance filtering excluded half of all reported effects, guaranteeing minimum effect sizes.
  • The mean correlation of filtered effects was overestimated by 56% compared to as-reported effects.
  • 92% of as-reported effects were smaller than the mean of filtered effects.

Conclusions:

  • Outcome-dependent selection of effects is circular and predetermines results.
  • Significance filtering contradicts the purpose of meta-analysis by inflating effect sizes.
  • Meta-analyses should adhere to established research practices, avoiding significance-based filtering and scoring.