Distributional Diagnosis and Calibration with Negative Controls for Outcome-wide Real-world Evidence

Area of Science:

• Real-world evidence synthesis
• Comparative effectiveness research (CER)
• Causal inference in observational studies

Background:

• Glucagon-like peptide-1 receptor agonists (GLP-1RAs) exhibit pleiotropic effects, necessitating comprehensive outcome-wide risk assessment.
• Residual bias after confounder adjustment in real-world data analyses can distort treatment effect estimates and uncertainty calibration.
• Existing causal inference methods struggle to fully address unmeasured confounding in large-scale observational studies.

Purpose of the Study:

• To introduce and validate distributional diagnosis and calibration (DC), a novel methodology for diagnosing and correcting residual bias in real-world CER.
• To assess the impact of DC on the reliability and calibration of comparative risk profiles for GLP-1RAs versus sodium-glucose cotransporter 2 inhibitors (SGLT2is).
• To demonstrate the utility of DC in strengthening the credibility of outcome-wide analyses using electronic health records.

Main Methods:

• Developed distributional diagnosis and calibration (DC), a modular approach utilizing negative control outcomes (NCOs) to detect and adjust for residual bias.
• Evaluated DC's performance through p-value uniformity and empirical coverage across NCOs, and calibrated confidence intervals for primary outcomes.
• Applied DC to a large electronic health record dataset (152.7 million patients), comparing GLP-1RAs with SGLT2is across 15 outcomes using four causal estimators.

Main Results:

• DC diagnostics revealed substantial and method-dependent residual systematic error across various causal inference methods.
• DC calibration effectively attenuated systematic error signals identified in NCOs, leading to more stable and reliable estimates.
• The calibrated estimates for clinical outcomes demonstrated improved accuracy and better calibration compared to uncalibrated estimates.

Conclusions:

• Distributional diagnosis and calibration (DC) is a practical and effective strategy for enhancing the credibility of outcome-wide real-world comparative effectiveness research.
• DC provides a robust framework for diagnosing residual bias and calibrating uncertainty, crucial for interpreting observational study findings.
• The methodology supports collaborative research by operating on summary statistics, facilitating broader application in real-world data analyses.