A Cautionary Note for Plasmode Simulation Studies in the Setting of Causal Inference

Area of Science:

• Pharmacoepidemiology
• Biostatistics
• Health Informatics

Background:

• Plasmode simulation is vital for assessing statistical methods in complex pharmacoepidemiological studies using electronic health records (EHR).
• These simulations help understand how estimator performance is affected by real-world data challenges like rare events and small sample sizes.
• Plasmode simulation combines real-world data resampling with probabilistic models to establish a known truth for estimands.

Purpose of the Study:

• To compare two prevalent plasmode simulation frameworks for evaluating methods that estimate a point treatment estimand.
• To identify potential biases introduced by different plasmode strategies in statistical analyses.
• To provide guidance on selecting appropriate plasmode frameworks for reliable method evaluation.

Main Methods:

• Comparative analysis of two distinct plasmode simulation frameworks.
• Development of numerical evidence and a theoretical result to demonstrate framework-induced bias.
• Extensive simulation studies using both model-generated and real-world EHR data, including scenarios with large sample sizes and randomized controlled trial data.

Main Results:

• One plasmode framework was found to cause certain estimators to appear erroneously biased.
• These identified pitfalls persist even with large sample sizes and when analyzing data from randomized controlled trials.
• The study provides empirical and theoretical evidence of framework-specific biases in statistical method evaluation.

Conclusions:

• The choice of plasmode simulation framework significantly impacts the perceived performance of statistical estimators.
• Frameworks that do not maintain good theoretical properties can lead to unfair evaluations of statistical methods.
• Guidance is provided for selecting plasmode frameworks that ensure accurate method assessment while preserving data realism.