Pooling, meta-analysis, and the evaluation of drug safety

BACKGROUND: The "integrated safety report" of the drug registration files submitted to health authorities usually summarizes the rates of adverse events observed for a new drug, placebo or active control drugs by pooling the safety data across the trials. Pooling consists of adding the numbers of events observed in a given treatment group across the trials and dividing the results by the total number of patients included in this group. Because it considers treatment groups rather than studies, pooling ignores validity of the comparisons and is subject to a particular kind of bias, termed "Simpson's paradox." In contrast, meta-analysis and other stratified analyses are less susceptible to bias. METHODS: We use a hypothetical, but not atypical, application to demonstrate that the results of a meta-analysis can differ greatly from those obtained by pooling the same data. In our hypothetical model, a new drug is compared to 1) a placebo in 4 relatively small trials in patients at high risk for a certain adverse event and 2) an active reference drug in 2 larger trials of patients at low risk for this event. RESULTS: Using meta-analysis, the relative risk of experiencing the adverse event with the new drug was 1.78 (95% confidence interval [1.02; 3.12]) compared to placebo and 2.20 [0.76; 6.32] compared to active control. By pooling the data, the results were, respectively, 1.00 [0.59; 1.70] and 5.20 [2.07; 13.08]. CONCLUSIONS: Because these findings could mislead health authorities and doctors, regulatory agencies should require meta-analyses or stratified analyses of safety data in drug registration files.