Generalizability analysis for clinical trials: a simulation study.

Area of Science:

• Biostatistics
• Clinical Trial Design
• Health Services Research

Background:

• Clinical trial participants are rarely randomly selected from target populations, leading to underrepresentation of key demographics like women, children, elderly, and those with comorbidities.
• Current practice often assumes trial findings are applicable to individual patients by using efficacy and assuming patient similarity.

Purpose of the Study:

• To investigate how nonrandom patient selection in clinical trials can bias results.
• To evaluate the performance of generalization indexes under varying treatment effect bias.

Main Methods:

• Simulated a patient population with a treatment effect size of 0.5 (Cohen's d) and five covariates: gender, health insurance, comorbidity, age, and motivation.
• Created 50 nonrandom clinical trials from this population to assess bias.
• Calculated and evaluated C-statistics, standardized mean difference (SMD), and Tipton's index (β) for generalization.

Main Results:

• Relative bias in nonrandom trials ranged from 1.68% to 99.70%.
• Generalization indexes showed wide ranges: C-statistics (0.56-0.98), SMD (0.23-11.17), and β (0.99-0.73) as bias increased.
• Thresholds for acceptable bias (<50%) were identified: C-statistics < 0.86, SMD < 1.95, and β > 0.91.

Conclusions:

• Nonrandom patient selection significantly biases clinical trial results and impacts generalizability.
• Existing generalization indexes (C-statistics, SMD, β) provide valuable metrics for assessing bias.
• Recommendations are provided for using these indexes to evaluate the generalizability of trial findings based on simulation results.