Related Experiment Video
Updated: Jan 23, 2026
Meta-Analysis of the Effectiveness and Safety of Shugan Jieyu Capsules for the Treatment of Insomnia
Published on: February 17, 2023
Apparently conclusive meta-analyses on interventions in critical care may be inconclusive-a meta-epidemiological
Thijs M Koster1, Jørn Wetterslev2, Christian Gluud2
1Department of Critical Care, University Medical Center Groningen, University of Groningen, Groningen, The Netherlands.
Objectives:
Risks of random type I and II errors are associated with false positive and false negative findings. In conventional meta-analyses, the risks of random errors are insufficiently evaluated. Many meta-analyses, which appear conclusive, might, in fact, be inconclusive because of risks of random errors. We hypothesize that, for interventions in critical care, false positive and false negative findings frequently become apparent when accounting for the risks of random error. We aim to investigate to which extent apparently conclusive conventional meta-analyses remain conclusive when adjusting statistical significance levels and confidence intervals considering sparse data and repeated testing through Trial Sequential Analysis (TSA).
Methods:
We searched The Cochrane Library, MEDLINE, and EMBASE for reviews of interventions in critical care. We used TSA with the relative risk reduction from the estimated meta-analyzed intervention effects adjusted for heterogeneity based on the observed diversity. We report proportions of meta-analyses and potential inconclusive findings of positive, neutral, and negative conclusions based on conventional naïve meta-analyses, which use an alpha of 5% and 95% confidence intervals. In TSA-controlled meta-analyses showing a beneficial or harmful intervention effect, we assessed the risk of bias by six Cochrane domains.
Results:
A total of 464 reviews containing 1,080 meta-analyses of (co-)primary outcomes were analyzed. From the 266 conventional meta-analyses suggesting a beneficial effect, 133 (50%) were true positive and 133 (50%) were potentially false positive according to TSA. From the 755 conventional meta-analyses suggesting a neutral effect, there were 214 (28%) true neutral and 541 (72%) were potentially false neutral according to TSA. From the 59 conventional meta-analyses suggesting a harmful effect, 17 (29%) were true negative and 42 (71%) were potentially false negative according to TSA. When the true beneficial and true harmful meta-analyses according to TSA were evaluated for risk of bias, new TSAs conducted on only trials with overall low risk of bias showed only firm evidence of a beneficial effect on one outcome and a harmful effect on one outcome.
Conclusions:
Of all meta-analyses in critical care, a large proportion may reach false conclusions because of unknown risks of random type I or type II errors. Future critical care meta-analyses should aim for establishing an effect of interventions accounting for risks of bias and random errors.
Insights
Many critical care meta-analyses may reach false conclusions due to random errors. Trial Sequential Analysis (TSA) reveals that 50% of beneficial findings and 71% of harmful findings were potentially false, highlighting the need for robust error assessment.
Area of Science:
- Critical care medicine
- Biostatistics
- Evidence-based medicine
Background:
- Conventional meta-analyses often inadequately assess risks of random errors (Type I and Type II), potentially leading to false positive or negative findings.
- Apparent conclusions from meta-analyses may be unreliable due to insufficient evaluation of random error risks.
Purpose of the Study:
- To investigate the extent to which conventional meta-analyses in critical care remain conclusive when accounting for random error risks.
- To evaluate the impact of adjusting statistical significance and confidence intervals using Trial Sequential Analysis (TSA) on meta-analysis conclusions.
Main Methods:
- Searched Cochrane Library, MEDLINE, and EMBASE for critical care intervention reviews.
- Applied TSA to meta-analyzed intervention effects, adjusting for heterogeneity and using relative risk reduction.
- Assessed risk of bias in TSA-controlled meta-analyses with significant findings using six Cochrane domains.
Main Results:
- Analyzed 1,080 meta-analyses from 464 reviews. 50% of meta-analyses suggesting benefit were potentially false positive; 72% of neutral findings and 71% of harmful findings were potentially false.
- After TSA adjustment, only one outcome showed firm evidence of benefit and one of harm when considering only low-risk-of-bias trials.
Conclusions:
- A significant proportion of critical care meta-analyses may yield false conclusions due to unaddressed random errors.
- Future critical care meta-analyses must incorporate methods to account for bias and random errors to establish reliable intervention effects.
More Related Videos
06:26Meta-analysis of Voxel-Based Neuroimaging Studies using Seed-based d Mapping with Permutation of Subject Images SDM-PSI
Published on: November 27, 2019
08:36Author Spotlight: Evaluating the Adjuvant Efficacy and Safety of Angong Niuhuang Pill in Viral Encephalitis Treatment
Published on: April 19, 2024
Related Concept Videos
01:09Directing Effect of Substituents: meta-Directing Groups
01:13meta-Directing Deactivators: –NO2, –CN, –CHO, –CO2R, –COR, –CO2H
01:20Study Designs in Epidemiology
Observational studies are those where the researcher does not intervene but rather observes natural variations. They include cross-sectional, cohort, and...
01:27Confounding in Epidemiological Studies
01:29Bias in Epidemiological Studies
01:26Introduction to Epidemiology