A Cautionary Note for Plasmode Simulation Studies in the Setting of Causal Inference

Plasmode simulation has become an important tool for evaluating operating characteristics of different statistical methods in complex settings, such as pharmacoepidemiological studies of treatment effectiveness using electronic health records (EHR) data. These studies provide insight into how estimator performance is impacted by challenges including rare events, small sample size and so forth that can indicate which among a set of methods performs best in a real-world dataset. Plasmode simulation combines data resampled from a real-world dataset with data from a probabilistic model to generate a known truth for an estimand in realistic data. There are different potential plasmode strategies currently in use. We compare two popular plasmode simulation frameworks for the evaluation of methods estimating a point treatment estimand. We provide numerical evidence and a theoretical result that shows one of these frameworks can cause certain estimators to incorrectly appear overly biased. Detailed simulation studies using both model-generated and real-world EHR data demonstrate these pitfalls remain at large sample sizes and when analyzing data from a randomized controlled trial. We conclude with guidance for the choice of a plasmode framework that maintains good theoretical properties to allow a fair evaluation of statistical methods while also maintaining the desired similarity to real data.