Diverse criteria and methods are used to compare treatment effect estimates: a scoping review

To determine what criteria researchers use to assess whether the estimates of effect of an intervention on a dichotomous outcome are different when obtained using different study designs.

Scoping review of the literature. We included studies of dichotomous outcomes in which authors compared the estimates of effects from different study designs. We performed searches in electronic databases and in the list of references of relevant studies. Two reviewers independently selected studies and abstracted data. We created a list of the criteria used to compare estimates of effects between study designs, described their main features, and classified them using a clinical perspective.

We included 26 studies, from which we identified 24 criteria. Most of the studies focused on comparing estimates from observational studies and randomized controlled trials (n = 19). The most common criteria aimed to determine whether there was a difference or not (n = 18), provided guidance for such a judgment (n = 16), and were based on the point estimates (n = 11). We judged 14 criteria to be appropriate and classified them as either statistically related or clinically related.

We found that diverse criteria are used to compare effect estimates between study designs. Familiarity with these would aid in the interpretation of results from different studies regarding the same question.