Trial sequential analysis may be insufficient to draw firm conclusions regarding statistically significant treatment...

Trial sequential analysis (TSA) has been proposed as a method to assess the risk of random error in cumulative meta-analysis (MA), which increases due to repeated significance testing. The aim of TSA is to assist researchers from wrongly concluding treatment differences in the absence of a benefit (i.e. true versus false positive). Similar to monitoring boundaries applied in individual randomized controlled trials, recent literature has advocated the use of TSA for assessing the conclusiveness of results from MAs to determine the requirement for future studies in case of true positive results. While this may be desirable, we present empirical evidence from a recent systematic review to demonstrate that the use of TSA may lead to a premature declaration of statistically significant treatment difference, when further accumulated evidence suggested otherwise. Using all apparently conclusive MAs in multiple-myeloma, we empirically studied under what thresholds for the risk ratio reduction and power a true positive result becomes false positive. We recommend that the conclusion of significant treatment differences in cumulative MA should be weighed against acceptable thresholds regarding the type I error, power and apriori specified clinically meaningful treatment difference.