Computation of the distribution of model accuracy statistics in machine learning: Comparison between analytically...

Area of Science:

• Machine learning in healthcare
• Statistical modeling
• Predictive analytics

Background:

• Machine learning is increasingly used across scientific fields.
• Accurate evaluation of machine learning models requires critical assessment of performance metrics like sensitivity, specificity, and AUROC.
• Novel methods for evaluating model efficacy are essential.

Purpose of the Study:

• To propose and evaluate analytically derived distributions (ADDs) as a method for assessing machine learning model metrics.
• To compare ADDs with traditional simulation-based approaches.

Main Methods:

• A retrospective cohort study using the England National Health Services Heart Disease Prediction Cohort.
• Four machine learning models were evaluated: XGBoost, Random Forest, Artificial Neural Network, and Adaptive Boost.
• Model metrics and covariate gain statistics were derived using bootstrap simulation (N=10,000) and compared with ADDs.

Main Results:

• XGBoost demonstrated the best performance with the highest AUROC and aggregate score.
• Distributions from bootstrap simulation did not significantly deviate from normal distribution (Anderson-Darling test).
• ADDs yielded smaller standard deviations (SDs) than bootstrap simulations, with other distribution aspects remaining similar.

Conclusions:

• Analytically derived distributions (ADDs) provide a reproducible alternative to simulation-based methods like bootstrapping for evaluating model metrics.
• ADDs facilitate cross-study comparisons of model performance, enhancing scientific transparency and replicability.