False-positive and false-negative risks for individual multicentre trials in critical care

Area of Science:

• Medical Statistics
• Clinical Trial Analysis
• Evidence-Based Medicine

Background:

• Null hypothesis significance testing (NHST) is the predominant statistical inference framework in medical research.
• NHST relies on P-values and confidence intervals, which quantify evidence against a null hypothesis but do not indicate the probability of the hypothesis being true.
• False-positive risk (FPR) and false-negative risk (FNR) represent post-test probabilities regarding the hypothesis's truth, indicating the likelihood of a real effect.

Purpose of the Study:

• To calculate the false-positive risk (FPR) and false-negative risk (FNR) for individual multicentre trials in critical care.
• To assess the probability of a true effect existing or being absent, providing a more direct measure of evidence quality than traditional NHST metrics.

Main Methods:

• Calculated FPR or FNR for 53 individual multicentre trials in critical care.
• Assumed a pretest probability of 0.5 for the hypothesis being true to determine post-test risks.

Main Results:

• For trials reporting statistical significance, FPR ranged from 0.1% to 57.6%.
• For trials reporting non-significance, FNR ranged from 1.7% to 36.9%.
• Twenty-six of 47 non-significant trials (55.3%) showed strong evidence for the null hypothesis; others provided limited evidence. No clear P-value/FNR relationship was observed.

Conclusions:

• FPR and FNR exhibited significant variability, indicating substantial differences in the probability of real or absent treatment effects across trials.
• Few trials provided convincing evidence of a real treatment effect (1/26 significant trials), and nearly half of non-significant trials offered limited evidence of absence of effect.
• The findings highlight highly variable evidence quality from multicentre trials in critical care, underscoring limitations of traditional NHST.