Warnings on the inclusion of cluster randomized trials in meta-analysis: results of a simulation study

Consolidation of treatment effects from randomized controlled trials (RCT) is considered one of the highest forms of evidence in research. Cluster randomized trials (CRT) are increasingly used in the assessment of the effectiveness of interventions when individual-level randomization is impractical. In meta-analyses, CRTs that address the same clinical question as RCTs can be pooled in the same analysis; however, they need to be analyzed with appropriate statistical methods. This study examined the extent to which meta-analysis results are influenced by the inclusion of incorrectly analyzed CRTs through a series of simulations.

RCT and CRT datasets were generated with a continuous treatment effect of zero, two trial arms, and equal number of participants. CRT datasets were generated with varying number of clusters (10, 20 or 40), observations per cluster (10, 30 or 50), total variance (1, 5 or 10) and ICC (0.05, 0.10 or 0.20). Each simulated CRT dataset (n = 1000 for each scenario) was analyzed using standard linear regression and mixed-effects regression with clusters treated as random effects to represent the incorrectly and correctly analyzed CRTs. Meta-analytic datasets were created by varying the total number of studies (4, 8 or 12), number of CRTs out of the total number of studies (single, half or all), and the number of correctly analyzed CRTs (none, half or all). Model performance was summarized from 1000 random-effects meta-analyses for each scenario.

The percentage of statistically significant results (at p < 0.05) was consistently lower when all CRTs were correctly analyzed. The alpha threshold (5%) was exceeded in 6 (2.47%) of 243 scenarios when all CRTs were correctly analyzed, compared to 177 (72.84%) and 195 (80.25%) scenarios when half or none of the CRTs were correctly analyzed, respectively. Coverage probabilities and model-based SEs were higher when all CRTs were correctly analyzed, while the estimated effect sizes and bias averaged across iterations showed no differences regardless of the number of correctly analyzed CRTs.

Ignoring clustering in CRTs lead to inflated false-positive conclusions about the efficacy of treatments, highlighting the need for caution and proper analytical methods when incorporating CRTs into meta-analyses.