Evaluating the Generalisability of Trial Results: Introducing a Centre- and Trial-Level Generalisability Index

Few randomised controlled trials (RCTs) recruit centres representatively, which may limit the external validity of trial results.

The aim of this study was to propose a proof-of-concept method of assessing the generalisability of the clinical and cost-effectiveness findings of a given RCT.

We developed a generalisability index (Gix), informed by centre-level characteristics, as a measure of centre and trial representativeness. The centre-level Gix quantifies how representative a centre is in relation to its jurisdiction, e.g. a country or health authority. The trial-level Gix quantifies how representative trial recruitment is in relation to clinical practice in the jurisdiction. Taking a real-world RCT as a case study and assuming trial-wide results to represent 'true jurisdiction values', we used simulation methods to recreate 5000 RCTs and investigate the relationship between trial representativeness, reflected by the standardised trial-Gix, and the deviation of simulated trial results from the 'true values'.

The simulation study provides evidence that trial results (odds ratio for the primary outcome and incremental quality-adjusted life-years) were influenced by the representativeness of the sample of recruiting centres. Simulated RCTs with the closest results to the 'true values' were those whose recruitment closely mirrored the jurisdiction-wide context. Results appeared robust to six alternative specifications of the Gix.

Our findings suggest that an unrepresentative selection of centres limits the external validity of trial results. The Gix may be a valuable tool to help facilitate rational selection of trial centres and ensure the generalisability of results at the jurisdiction level.