I2 Statistic as the Selection Bias Test: Trial Effect Estimates in Relation to Identified Bias Levels

This study aimed to investigate the association between selection bias, established by the use of the I² test from published randomised controlled trials (RCTs), and the effect estimate magnitude of these trials. Two null hypotheses were tested: H01: The magnitude of trial effect estimates is not significantly positively correlated with the identified selection bias levels. H02: The magnitude of trial effect estimates does not differ significantly between RCTs with identified 'low' and 'high' selection biases.

RCTs reporting computable outcomes and baseline data were selected from published systematic review reports that, in turn, were identified through a systematic literature search in PubMed up to 2024. All RCTs were tested for selection bias using the trial-adjusted, simulated comparator trial (SCT)-based I² test. For each RCT, the selection bias level (B%) was determined, and the absolute value of the risk difference (RD) point estimate was calculated. H01 was tested using Spearman's rank correlation, and H02 was tested using an independent samples t-test. A sensitivity analysis was carried out to examine any potential confounder effect.

A total of 332 RCTs, published from 1985 to 2023 in various medical specialties, were tested. Test applicability was limited by the low quality of RCT reporting. 'Low' selection bias was identified in 202 RCTs and 'high' selection bias in 130 RCTs. The estimation of selection bias levels within pre-specified I² point estimate thresholds was possible for 71% of all RCTs. For 97 (29%) RCTs, the computed I² point estimates fell outside these thresholds, and therefore, B% estimation was possible by approximation only. Such an approximation proved to have a confounding effect. After confounder correction, there was a significant positive correlation between the magnitudes of trial effect estimates and the selection bias levels (Spearman's r = 0.25, p < 0.001). The effect estimates were statistically significantly higher (0.07; 95%CI: 0.03-0.11; p = 0.0005) for RCTs with identified 'high' selection bias than for RCTs with 'low' selection bias, representing a proportional over-estimation of 64%. Both null hypotheses H01 and H02 were rejected.

The trial-adjusted, SCT-based I² test appeared to be effective for identifying high-level selection bias in RCTs. The test may allow estimation of the selection bias extent and its possible effect on the reported trial effect estimate.