Generalizability analysis for clinical trials: a simulation study

Subjects are rarely selected on a random basis from a well-defined patient population of interest into a clinical trial, with women, children, the elderly, and those with common comorbidities who are frequently underrepresented. Decades of clinical experience have demonstrated that the application of trial findings to individual patients is permissible by using efficacy as a measure of effectiveness and assuming that the characteristics of patients are sufficiently similar. In order to investigate this issue in greater depth, we simulated a patient population with treatment effect size of 0.5 (Cohen's d) and five covariates that included gender, health insurance, comorbidity, age, and motivation. To demonstrate how selection of patients for a clinical trial can bias the results when treatment effect varies across individuals, we created 50 nonrandom clinical trials based on this patient population and showed relative bias to range from 1.68% to 99.70%. We calculated and evaluated three indexes: C-statistics, standardized mean difference (SMD), and Tipton's index (β) of generalization for the 50 nonrandom trials. Findings indicated that (i) the ranges were 0.56-0.98, 0.23-11.17, and 0.99-0.73 for C-statistics, SMD, and β, respectively, when treatment effect bias increased from 1.68% to 99.70% and (ii) C-statistics < 0.86, SMD < 1.95, and β > 0.91 when treatment effect bias <50%. Recommendations are made using existing generalization indexes on the basis of our simulation results. An example from a real clinical trial is provided for illustration. Copyright © 2017 John Wiley & Sons, Ltd.