Related Experiment Video
Updated: Jun 8, 2025
An R-Based Landscape Validation of a Competing Risk Model
Published on: September 16, 2022
Unraveling Uncertainty: The Impact of Biological and Analytical Variation on the Prediction Uncertainty of
Remy J H Martens1,2, William P T M van Doorn2, Mathie P G Leers1,3
1Department of Clinical Chemistry and Hematology, Zuyderland Medical Center, Sittard-Geleen, the Netherlands.
Background:
Interest in prediction models, including machine learning (ML) models, based on laboratory data has increased tremendously. Uncertainty in laboratory measurements and predictions based on such data are inherently intertwined. This study developed a framework for assessing the impact of biological and analytical variation on the prediction uncertainty of categorical prediction models.
Methods:
Practical application was demonstrated for the prediction of renal function loss (Chronic Kidney Disease Epidemiology Collaboration [CKD-EPI] equation) and 31-day mortality (advanced ML model) in 6360 emergency department patients. Model outcome was calculated in 100 000 simulations of variation in laboratory parameters. Subsequently, the percentage of discordant predictions was calculated with the original prediction as reference. Simulations were repeated assuming increasing levels of analytical variation.
Results:
For the ML model, area under the receiver operating characteristic curve (AUROC), sensitivity, and specificity were 0.90, 0.44, and 0.96, respectively. At base analytical variation, the median [2.5th-97.5th percentiles] percentage of discordant predictions was 0% [0%-28.8%]. In addition, 7.2% of patients had >5% discordant predictions. At 6× base analytical variation, the median [2.5th-97.5th percentiles] percentage of discordant predictions was 0% [0%-38.8%]. In addition, 11.7% of patients had >5% discordant predictions. However, the impact of analytical variation was limited compared with biological variation. AUROC, sensitivity, and specificity were not affected by variation in laboratory parameters.
Conclusions:
The impact of biological and analytical variation on the prediction uncertainty of categorical prediction models, including ML models, can be estimated by the occurrence of discordant predictions in a simulation model. Nevertheless, discordant predictions at the individual level do not necessarily affect model performance at the population level.
More Related Videos
Related Concept Videos
01:11Variability: Analysis
The range is a simple measure of variability, indicating the difference between the highest and...
00:59Uncertainty: Overview
01:19Variation
When independent and dependent variables are plotted on a scatter plot, the slope of a line is a value that describes the rate of change between the two...
01:23Mechanistic Models: Compartment Models in Individual and Population Analysis
00:59Propagation of Uncertainty from Random Error
01:11How Data are Classified: Categorical Data
Data are classified based on whether they are measurable or not. Categorical data cannot be measured; instead, it can be divided into categories. For example, if Y denotes a person's party affiliation, some examples of Y include...