Adjusting for both sequential testing and systematic error in safety surveillance using observational data: Empirical...

Area of Science:

• Pharmacovigilance
• Biostatistics
• Observational Data Analysis

Background:

• Post-approval safety surveillance using observational healthcare data is crucial for detecting medical product safety issues missed in pre-approval trials.
• Sequential testing methods like maximum sequential probability ratio testing (MaxSPRT) are used to maintain type 1 error rates as data accrue.
• Systematic errors inherent in observational data analyses can cause true type 1 error rates to deviate from nominal levels, even after controlling for confounders.

Purpose of the Study:

• To address the issue of systematic error in sequential safety surveillance of medical products.
• To propose and evaluate a novel approach combining MaxSPRT with empirical calibration to restore nominal type 1 error rates.
• To demonstrate the method's effectiveness using simulations and real-world electronic health records data.

Main Methods:

• Proposed a novel method integrating empirical calibration with MaxSPRT to address systematic error in sequential testing.
• Empirical calibration infers a probability distribution of systematic error using negative control exposure-outcome pairs.
• The approach calibrates the critical value central to MaxSPRT and was evaluated using simulations and H1N1 vaccination data.

Main Results:

• The combined approach of empirical calibration and MaxSPRT successfully restored nominal type 1 error rates in simulations and real-world data.
• In the H1N1 vaccination example, empirical calibration's impact on adjusting for systematic error was greater than MaxSPRT's impact from sequential testing.
• Both empirical calibration for systematic error and MaxSPRT for sequential testing are essential for reliable safety surveillance.

Conclusions:

• Combining empirical calibration with MaxSPRT is a robust method for accurate post-approval safety surveillance of medical products.
• Empirical calibration is as essential as sequential testing (MaxSPRT) for managing systematic errors in observational data.
• The described method is recommended for performing both systematic error adjustment and sequential testing in safety surveillance.