Related Experiment Video
Updated: Jun 5, 2025
08:36
Author Spotlight: Evaluating the Adjuvant Efficacy and Safety of Angong Niuhuang Pill in Viral Encephalitis Treatment
Published on: April 19, 2024
436
Evaluating meta-analysis as a replication success measure
Jasmine Muradchanian1, Rink Hoekstra1, Henk Kiers1
1Behavioural and Social Sciences, University of Groningen, Groningen, the Netherlands.
Plos One
|December 11, 2024
Summary
Meta-analysis performs poorly as a replication success metric in social sciences. It often incorrectly concludes success, regardless of replication results, due to original study significance.
Area of Science:
- Social and behavioral sciences
- Research methodology
- Statistical analysis
Background:
- Replication is crucial in social and behavioral sciences.
- Meta-analysis is one method to assess replication success.
- Previous methods for qualifying replication success exist.
Purpose of the Study:
- To evaluate meta-analysis as a metric for replication success.
- To investigate how meta-analysis functions in qualifying replication outcomes.
- To assess the accuracy of meta-analysis predictions of replication success.
Main Methods:
- Utilized original and replication studies from large-scale replication projects.
- Calculated the probability of replication success using meta-analysis.
- Assessed prediction accuracy with adjusted Brier scores after replication results were known.
Main Results:
- Meta-analysis performed poorly as a replication success metric.
- Conclusions of success were often reached irrespective of actual replication study findings.
- Meta-analysis showed a high probability of detecting a non-zero effect even when the true effect was zero.
Conclusions:
- Meta-analysis is not a reliable metric for replication success.
- Original study significance heavily influences meta-analysis outcomes.
- Fundamental issues exist with using meta-analysis to judge replication success.
More Related Videos
06:26Meta-analysis of Voxel-Based Neuroimaging Studies using Seed-based d Mapping with Permutation of Subject Images SDM-PSI
Published on: November 27, 2019
70.0K
11:58The Replica Set Method: A High-throughput Approach to Quantitatively Measure Caenorhabditis elegans Lifespan
Published on: June 29, 2018
9.4K
Related Concept Videos
01:11Statistical Analysis: Overview
6.1K
When we take repeated measurements on the same or replicated samples, we will observe inconsistencies in the magnitude. These inconsistencies are called errors. To categorize and characterize these results and their errors, the researcher can use statistical analysis to determine the quality of the measurements and/or suitability of the methods.
One of the most commonly used statistical quantifiers is the mean, which is the ratio between the sum of the numerical values of all results and the...
One of the most commonly used statistical quantifiers is the mean, which is the ratio between the sum of the numerical values of all results and the...
6.1K
01:13Accuracy and Errors in Hypothesis Testing
175
Hypothesis testing is a fundamental statistical tool that begins with the assumption that the null hypothesis H0 is true. During this process, two types of errors can occur: Type I and Type II. A Type I error refers to the incorrect rejection of a true null hypothesis, while a Type II error involves the failure to reject a false null hypothesis.
In hypothesis testing, the probability of making a Type I error, denoted as α, is commonly set at 0.05. This significance level indicates a 5%...
In hypothesis testing, the probability of making a Type I error, denoted as α, is commonly set at 0.05. This significance level indicates a 5%...
175
01:52Regression Toward the Mean
6.3K
Regression toward the mean (“RTM”) is a phenomenon in which extremely high or low values—for example, and individual’s blood pressure at a particular moment—appear closer to a group’s average upon remeasuring. Although this statistical peculiarity is the result of random error and chance, it has been problematic across various medical, scientific, financial and psychological applications. In particular, RTM, if not taken into account, can interfere when...
6.3K
01:50Statistical Significance
20.1K
Once data is collected from both the experimental and the control groups, a statistical analysis is conducted to find out if there are meaningful differences between the two groups. A statistical analysis determines how likely any difference found is due to chance (and thus not meaningful). In psychology, group differences are considered meaningful, or significant, if the odds that these differences occurred by chance alone are 5 percent or less. Stated another way, if we repeated this...
20.1K
01:15Data Validation
142
Method validation is a crucial process in analytical chemistry designed to confirm that a given method consistently produces reliable and high-quality results. This process is essential when a method is applied to different sample matrices or when procedural modifications are made, ensuring that the results meet acceptable standards across various applications.
Key parameters for method validation include:
Key parameters for method validation include:
142
01:29Reliability and Validity
12.7K
Reliability and validity are two important considerations that must be made with any type of data collection. Reliability refers to the ability to consistently produce a given result. In the context of psychological research, this would mean that any instruments or tools used to collect data do so in consistent, reproducible ways.
12.7K