Related Experiment Video
Updated: Mar 2, 2026
An R-Based Landscape Validation of a Competing Risk Model
Published on: September 16, 2022
Validation of a Cardiovascular Disease Policy Microsimulation Model Using Both Survival and Receiver Operating
Ankur Pandya1, Stephen Sy1, Sylvia Cho2
1Department of Health Policy and Management, Harvard T.H. Chan School of Public Health, Boston, MA, USA (AP, SS, SA, MCW, TAG).
Insights
A new cardiovascular disease (CVD) policy simulation model accurately predicts mortality. This validated model can identify cost-effective strategies for reducing the burden of CVD.
Area of Science:
- Public Health
- Epidemiology
- Health Policy
Background:
- Cardiovascular disease (CVD) remains a leading cause of death and high healthcare costs in the United States.
- Effective policy interventions are needed to mitigate the substantial burden of CVD.
- A comprehensive simulation model is required to identify cost-effective CVD reduction strategies.
Purpose of the Study:
- To detail the development of a novel cardiovascular disease (CVD) policy simulation model.
- To validate the accuracy of the CVD policy simulation model using real-world data.
- To assess the model's capability in predicting mortality outcomes.
Main Methods:
- A microsimulation model was developed, simulating 1,000,000 adults aged 35-80.
- The model incorporated CVD-related epidemiological data, including Framingham-based risk scores.
- Validation utilized National Health and Nutrition Examination Survey (NHANES) data (1999-2011) and compared simulated mortality with observed data using survival and ROC curves.
Main Results:
- The simulation model closely matched observed 10-year all-cause mortality (10.9% simulated vs. 11.2% observed) and CVD mortality (2.6% simulated vs. 2.2% observed).
- Receiver operating characteristic (ROC) curve analysis demonstrated strong model performance for predicting 10-year all-cause mortality (Area Under Curve [AUC] = 0.83) and CVD mortality (AUC = 0.84).
- The model also showed good discrimination for 5-year mortality risks (AUCs ranging from 0.80 to 0.81).
Conclusions:
- The developed CVD policy simulation model demonstrates robust performance in aligning with nationally representative longitudinal mortality data.
- Receiver operating characteristic (ROC) curve analysis is a valuable method for assessing the discrimination capabilities of disease simulation models.
- The validated model serves as a reliable tool for evaluating cost-effective CVD policy interventions.
Background:
Despite some advances, cardiovascular disease (CVD) remains the leading cause of death and healthcare costs in the United States. We therefore developed a comprehensive CVD policy simulation model that identifies cost-effective approaches for reducing CVD burden. This paper aims to: 1) describe our model in detail; and 2) perform model validation analyses.
Methods:
The model simulates 1,000,000 adults (ages 35 to 80 years) using a variety of CVD-related epidemiological data, including previously calibrated Framingham-based risk scores for coronary heart disease and stroke. We validated our microsimulation model using recent National Health and Nutrition Examination Survey (NHANES) data, with baseline values collected in 1999-2000 and cause-specific mortality follow-up through 2011. Model-based (simulated) results were compared to observed all-cause and CVD-specific mortality data (from NHANES) for the same starting population using survival curves and, in a method not typically used for disease model validation, receiver operating characteristic (ROC) curves.
Results:
Observed 10-year all-cause mortality in NHANES v. the simulation model was 11.2% (95% CI, 10.3% to 12.2%) v. 10.9%; corresponding results for CVD mortality were 2.2% (1.8% to 2.7%) v. 2.6%. Areas under the ROC curves for model-predicted 10-year all-cause and CVD mortality risks were 0.83 (0.81 to 0.85) and 0.84 (0.81 to 0.88), respectively; corresponding results for 5-year risks were 0.80 (0.77 to 0.83) and 0.81 (0.75 to 0.87), respectively.
Limitations:
The model is limited by the uncertainties in the data used to estimate its input parameters. Additionally, our validation analyses did not include non-fatal CVD outcomes due to NHANES data limitations.
Conclusions:
The simulation model performed well in matching to observed nationally representative longitudinal mortality data. ROC curve analysis, which has been traditionally used for risk prediction models, can also be used to assess discrimination for disease simulation models.
Related Concept Videos
01:15Assumptions of Survival Analysis
01:20Comparing the Survival Analysis of Two or More Groups
01:21Cancer Survival Analysis
01:20Actuarial Approach
Consider the example of a high-risk surgical procedure with significant early-stage mortality. A two-year clinical study is conducted,...
01:24Kaplan-Meier Approach
01:18Introduction To Survival Analysis
The primary goal of survival analysis is to estimate survival time—the time...