Robust Estimation of Loss-Based Measures of Model Performance under Covariate Shift

Area of Science:

• * Biostatistics
• * Machine Learning
• * Epidemiology

Background:

• * Prediction models are often developed in one population but applied to another.
• * Existing methods for performance estimation in target populations have limitations, especially with complex data structures.
• * Differences in covariate distributions between source and target populations can bias performance estimates.

Purpose of the Study:

• * To develop robust methods for estimating prediction model performance in a target population using limited data.
• * To enable the use of data-adaptive techniques for estimating nuisance parameters in performance estimation.
• * To provide reliable risk prediction in target populations distinct from the model's development source population.

Main Methods:

• * Development of novel, robust estimators for target population risk (expected loss).
• * Integration of data-adaptive methods, including machine learning, for nuisance parameter estimation.
• * Examination of large-sample properties and finite sample performance through simulations.
• * Application to lung cancer screening data from the National Health and Nutrition Examination Survey (NHANES).
• * Extension of methods to accommodate complex survey designs, such as NHANES.

Main Results:

• * Proposed estimators demonstrate robustness in performance estimation across differing populations.
• * Data-adaptive approaches enhance the reliability of nuisance parameter estimation.
• * Simulations confirm the effectiveness of the developed methods in finite samples.
• * Successful application to real-world lung cancer screening data highlights practical utility.

Conclusions:

• * The developed methods offer improved accuracy for prediction model performance assessment in target populations.
• * Robust estimators facilitate more reliable risk prediction, especially when populations diverge.
• * The approach is adaptable to complex survey data, broadening its applicability in public health research.