Related Experiment Video
Updated: May 30, 2025
12:18
A Machine Learning Approach to Design an Efficient Selective Screening of Mild Cognitive Impairment
Published on: January 11, 2020
7.4K
The Harms of Class Imbalance Corrections for Machine Learning Based Prediction Models: A Simulation Study
Alex Carriero1, Kim Luijken1, Anne de Hond1
1Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht, The Netherlands.
Statistics in Medicine
|January 27, 2025
Summary
Correcting for class imbalance in machine learning models may harm calibration. Models without imbalance correction consistently showed better or equal calibration, avoiding risk over-estimation in clinical prediction.
Area of Science:
- Machine Learning
- Biostatistics
- Clinical Informatics
Background:
- Risk prediction models are vital for clinical decisions, requiring accurate calibration.
- Healthcare data often exhibit class imbalance, leading researchers to apply corrections.
- The impact of these imbalance corrections on model calibration remains unclear.
Purpose of the Study:
- To investigate the effect of class imbalance corrections on the calibration of machine learning models.
- To compare calibration performance between models with and without imbalance correction across various scenarios.
Main Methods:
- Utilized extensive Monte Carlo simulations to assess out-of-sample predictive performance.
- Evaluated machine learning algorithms under different data-generating conditions (sample size, predictors, event fraction).
- Illustrated findings with a case study using MIMIC-III data.
Main Results:
- Models developed without class imbalance correction consistently demonstrated superior or equivalent calibration.
- Imbalance correction led to miscalibration, characterized by risk over-estimation.
- Re-calibration did not always resolve the miscalibration introduced by imbalance correction.
Conclusions:
- Class imbalance correction is not universally required for clinical prediction models.
- Applying imbalance correction can potentially impair model calibration and reliability.
- Prioritize model calibration over automatic imbalance correction for individual risk estimation.
Related Concept Videos
01:19Survival Tree
55
Survival trees are a non-parametric method used in survival analysis to model the relationship between a set of covariates and the time until an event of interest occurs, often referred to as the "time-to-event" or "survival time." This method is particularly useful when dealing with censored data, where the event has not occurred for some individuals by the end of the study period, or when the exact time of the event is unknown.
Building a Survival Tree
Constructing a...
Building a Survival Tree
Constructing a...
55
01:03Prediction Intervals
2.2K
The interval estimate of any variable is known as the prediction interval. It helps decide if a point estimate is dependable.
However, the point estimate is most likely not the exact value of the population parameter, but close to it. After calculating point estimates, we construct interval estimates, called confidence intervals or prediction intervals. This prediction interval comprises a range of values unlike the point estimate and is a better predictor of the observed sample value, y.
However, the point estimate is most likely not the exact value of the population parameter, but close to it. After calculating point estimates, we construct interval estimates, called confidence intervals or prediction intervals. This prediction interval comprises a range of values unlike the point estimate and is a better predictor of the observed sample value, y.
2.2K
01:52Regression Toward the Mean
6.3K
Regression toward the mean (“RTM”) is a phenomenon in which extremely high or low values—for example, and individual’s blood pressure at a particular moment—appear closer to a group’s average upon remeasuring. Although this statistical peculiarity is the result of random error and chance, it has been problematic across various medical, scientific, financial and psychological applications. In particular, RTM, if not taken into account, can interfere when...
6.3K
01:23Mechanistic Models: Compartment Models in Individual and Population Analysis
26
Mechanistic models are utilized in individual analysis using single-source data, but imperfections arise due to data collection errors, preventing perfect prediction of observed data. The mathematical equation involves known values (Xi), observed concentrations (Ci), measurement errors (εi), model parameters (ϕj), and the related function (ƒi) for i number of values. Different least-squares metrics quantify differences between predicted and observed values. The ordinary least...
26
01:13Accuracy and Errors in Hypothesis Testing
171
Hypothesis testing is a fundamental statistical tool that begins with the assumption that the null hypothesis H0 is true. During this process, two types of errors can occur: Type I and Type II. A Type I error refers to the incorrect rejection of a true null hypothesis, while a Type II error involves the failure to reject a false null hypothesis.
In hypothesis testing, the probability of making a Type I error, denoted as α, is commonly set at 0.05. This significance level indicates a 5%...
In hypothesis testing, the probability of making a Type I error, denoted as α, is commonly set at 0.05. This significance level indicates a 5%...
171
02:16Stereotype Content Model
14.0K
The Stereotype Content Model (SCM) was first proposed by Susan Fiske and her colleagues (Fiske, Cuddy, Glick & Xu, 2002; see also Fiske, 2012 and Fiske, 2017). The SCM specifies that when someone encounters a new group, they will stereotype them based on two metrics: warmth—or that group’s perceived intent, and how likely they are to provide help or inflict harm—and competence—or their ability to carry out that objective. Depending on the warmth-competence...
14.0K