Using public clinical trial reports to probe non-experimental causal inference methods.

Area of Science:

• Biostatistics
• Observational Study Design
• Clinical Trial Analysis

Background:

• Non-experimental studies are crucial for medical intervention effect estimation but difficult to evaluate due to untestable assumptions.
• Lack of verifiability hinders comparison and trust in observational study methods and their results.
• Existing methods for evaluating non-experimental studies are limited, necessitating new approaches for reliable assessment.

Purpose of the Study:

• To introduce TrialProbe, a novel data resource and statistical framework for evaluating non-experimental study methods.
• To establish a benchmark for assessing the reliability of observational study designs in medical research.
• To provide a method for comparing different non-experimental approaches against a validated standard.

Main Methods:

• Collected pseudo 'ground truths' on drug effects by analyzing adverse events from clinical trial reports using empirical Bayesian techniques.
• Developed a framework to evaluate non-experimental methods by measuring concordance between their effect estimates and clinical trial results.
• Demonstrated the approach by comparing propensity score matching, inverse propensity score weighting, and an unadjusted method on insurance claims data.

Main Results:

• Extracted 12,967 unique drug/adverse event comparisons from 33,701 clinical trial records to form a ground truth set.
• Propensity score matching and inverse propensity score weighting demonstrated high concordance with clinical trial results.
• Both propensity score matching and inverse propensity score weighting substantially outperformed an unadjusted baseline approach.

Conclusions:

• TrialProbe effectively probes non-experimental study methods by generating large ground truth sets.
• The framework distinguishes the performance of non-experimental methods in real-world observational data.
• TrialProbe enhances the reliability and trustworthiness of findings from observational medical studies.