Reporting Quality of Discrete Event Simulations in Healthcare-Results From a Generic Reporting Checklist

The aims of this study were to formulate a generic reporting checklist for healthcare-related discrete event simulation (DES) studies and to critically appraise the existing studies.

Based on the principles of accessibility and generality, assessment items were derived from the International Society for Pharmacoeconomics and Outcomes Research (ISPOR)-Society for Medical Decision Making (SMDM) Task Force reports. The resulting checklist was applied to all 211 DES studies identified in a previous review. The proportion of fulfilled checklist items served as an indicator of reporting quality. A logistic regression was conducted to investigate whether study characteristics (eg, publication before or after the publication of the ISPOR-SMDM reports) increased the likelihood of fulfilling more than the mean number of items fulfilled by the appraised DES studies.

An 18-item checklist was formulated covering model conceptualization, parameterization and uncertainty assessment, validation, generalizability, and stakeholder involvement. The reporting quality of the DES models fluctuated around the mean of 63.7% (SD 11.0%) over the period studied. A modest nonsignificant improvement in reporting quality was found after the publication of the ISPOR-SMDM reports (64.5% vs 62.9%). Items with the lowest performance were related to predictive validation (2.8% of studies), cross validation (8.5%), face validity assessment (26.5%), and stakeholder involvement (27.5%). Models applied to health economic evaluation (HEE), country under study, and industry sponsorship were significantly associated with the odds of achieving above-average reporting quality.

The checklist is applicable across various model-based analyses beyond HEEs. Adherence to the ISPOR-SMDM guidelines should be improved, particularly regarding model validation.