Evaluating performance of risk identification methods through a large-scale simulation of observational data

Area of Science:

• Observational healthcare data analysis
• Statistical methodology for drug safety
• Health informatics and simulation modeling

Background:

• Limited evaluation exists for statistical methods identifying drug safety risks in observational healthcare data.
• Existing simulations often fail to capture real-world complexities challenging observational analyses.

Purpose of the Study:

• To develop and assess OSIM2, a probabilistic framework for generating simulated observational healthcare data.
• To utilize this simulated data for evaluating methods that identify drug exposure-health outcome associations.

Main Methods:

• Generated longitudinal data for 10 million patients using a model from administrative claims.
• Applied seven observational designs to 399 drug-outcome scenarios with varying relative risks (RR).
• Validated simulations via descriptive comparison with real data; evaluated methods using AUC, bias, and MSE.

Main Results:

• OSIM2 accurately replicates confounding patterns found in real claims data.
• All designs showed good predictive accuracy (AUC > 0.90) for RR ≥ 2, but performance declined for RR < 2.
• Method bias profiles varied and were dependent on effect size.

Conclusions:

• OSIM2 serves as a valuable tool for methodological research in drug safety.
• Simulation results indicate that the operating characteristics of statistical methods differ from their theoretical properties.