Effective sample size for individual risk predictions: quantifying uncertainty in machine learning models | JoVE Visualize

Area of Science:

Clinical prediction modeling
Machine learning in healthcare
Statistical uncertainty quantification

Background:

Standard performance metrics for clinical prediction models do not adequately capture individual prediction uncertainty.
This lack of uncertainty assessment raises concerns about fairness, as models may be more certain for some patients than others.
Effective sample size has been proposed as a metric to quantify sampling uncertainty.

Purpose of the Study:

To develop and illustrate a computational method for estimating effective sample sizes across diverse prediction models.
To assess the utility of effective sample size in understanding individual prediction uncertainty in a large clinical dataset.
To explore the implications of effective sample size for communicating risk prediction uncertainty.

Main Methods:

A computational method was developed to estimate effective sample sizes for various prediction models, including logistic regression, elastic net, XGBoost, neural network, and random forest.
The method was applied to a clinical dataset comprising 23,034 individuals.
Simulations were conducted to evaluate the accuracy of the effective sample size estimates for different model types.

Main Results:

The developed method accurately estimated effective sample sizes for logistic regression and elastic net models, with minor deviations for XGBoost, neural network, and random forest.
Despite similar overall model performance metrics, substantial variations in effective sample sizes and patient-specific risk predictions were observed.
Individual prediction uncertainty was found to be significant, even when models were trained on large sample sizes.

Conclusions:

Individual prediction uncertainty in clinical models can be substantial, irrespective of the dataset size.
Effective sample size is a valuable measure for quantifying and communicating the uncertainty associated with individual risk predictions.
This approach holds promise for improving the transparency and fairness of machine learning-based prediction models in clinical practice.