Two-stage sampling designs for external validation of personal risk models

Area of Science:

• Epidemiology
• Biostatistics
• Health Informatics

Background:

• Validating personal risk models with right-censored cohort data presents challenges, including competing risk analysis and high costs for rare outcomes.
• Traditional full cohort validation requires extensive covariate assembly and risk assignment, especially when biological specimens are involved.

Purpose of the Study:

• To propose a cost-effective sampling design and estimating procedure for validating personal risk models using right-censored cohort data.
• To address complications of competing risks and the expense of full cohort validation for rare outcomes.

Main Methods:

• A two-stage sampling design is introduced, focusing on informative subjects for parameter estimation.
• Methods for estimating outcome probabilities and their variances (theoretical and bootstrap) are provided.
• Guidance on selecting two-stage designs to minimize efficiency loss is described.

Main Results:

• The proposed two-stage design efficiently validates risk models, particularly for rare outcomes and right-censored data.
• Optimal designs for one performance parameter may not be optimal for others, necessitating trade-offs.
• A practical design sampling all outcome-positive and more outcome-negative than censored subjects performs well generally.

Conclusions:

• The developed two-stage sampling design offers a cost-effective solution for validating personal risk models.
• The methods are robust to competing risks and reduce the burden of covariate analysis.
• The Risk Model Assessment Program (R package) implements these validated methods.