Measuring representation in clinical trials: Simulations demonstrating how current methods fail in the context of...

Clinical trial representativeness is vital for the evaluation of intervention performance and study generalizability. Current methods for evaluating representativeness are limited by the quality of the disease registry data used and may not appropriately evaluate studies aimed at precision cohorts. This study evaluates the sensitivity of existing methods for measuring study representation to differences between the population targeted by the clinical intervention and the closest available population in a patient registry. Using records for U.S.-based cancer clinical trials registered to ClinicalTrials.gov from 2017-2023 and the U.S. Cancer Statistics Public Use Database we calculated representativeness measures by comparing the demographic mix (based on sex, race, and ethnicity) in each clinical trial to the demographic mix for the same form of cancer in the U.S. Cancer Statistics database via exact Binomial tests. Then, the same tests were conducted comparing trial populations to simulated populations that were demographically different from the U.S. Cancer Statistics data by fixed percentages. The outcome of interest was whether the result of the test changed when the comparator population was different. For clinical trials reporting the sex, race, and ethnicity of participants, 24, 40, and 32 percent of studies (respectively) give different results when the difference between the registry population and simulated population is 5 percentage points. For all demographics, larger differences between the registry and the simulated populations were associated with worse metric performance. Analyses of clinical trial representativeness suffer from a large loss of accuracy in settings where treatments are targeted to demographically different subgroups. Studies of representation in the context of precision medicine should be interpreted with caution.