Related Experiment Video
Updated: Sep 27, 2025

An R-Based Landscape Validation of a Competing Risk Model
Published on: September 16, 2022
Quantitative prediction error analysis to investigate predictive performance under predictor measurement
Kim Luijken1, Jia Song2, Rolf H H Groenwold2,3
1Department of Clinical Epidemiology, Leiden University Medical Center, Leiden, the Netherlands. k.luijken@lumc.nl.
Background:
When a predictor variable is measured in similar ways at the derivation and validation setting of a prognostic prediction model, yet both differ from the intended use of the model in practice (i.e., "predictor measurement heterogeneity"), performance of the model at implementation needs to be inferred. This study proposed an analysis to quantify the impact of anticipated predictor measurement heterogeneity.
Methods:
A simulation study was conducted to assess the impact of predictor measurement heterogeneity across validation and implementation setting in time-to-event outcome data. The use of the quantitative prediction error analysis was illustrated using an example of predicting the 6-year risk of developing type 2 diabetes with heterogeneity in measurement of the predictor body mass index.
Results:
In the simulation study, calibration-in-the-large of prediction models was poor and overall accuracy was reduced in all scenarios of predictor measurement heterogeneity. Model discrimination decreased with increasing random predictor measurement heterogeneity.
Conclusions:
Heterogeneity of predictor measurements across settings of validation and implementation reduced predictive performance at implementation of prognostic models with a time-to-event outcome. When validating a prognostic model, the targeted clinical setting needs to be considered and analyses can be conducted to quantify the impact of anticipated predictor measurement heterogeneity on model performance at implementation.
Related Concept Videos
Statistical Analysis: Overview
One of the most commonly used statistical quantifiers is the mean, which is the ratio between the sum of the numerical values of all results and the...
Accuracy and Errors in Hypothesis Testing
In hypothesis testing, the probability of making a Type I error, denoted as α, is commonly set at 0.05. This significance level indicates a 5%...
Uncertainty in Measurement: Accuracy and Precision
Mechanistic Models: Compartment Models in Individual and Population Analysis
Variation
When independent and dependent variables are plotted on a scatter plot, the slope of a line is a value that describes the rate of change between the two...
Detection of Gross Error: The Q Test

