Increasing transparency in machine learning through bootstrap simulation and shapely additive explanations.

Area of Science:

• Medical informatics
• Machine learning in healthcare
• Biostatistics

Background:

• Machine learning (ML) models are increasingly used in medicine, but assessing their reliability and efficacy is challenging.
• Difficulty in evaluating ML models hinders researchers' ability to select appropriate models for specific datasets.
• Lack of transparency in ML model performance metrics complicates clinical adoption.

Purpose of the Study:

• To assess if variance calculations using bootstrap simulation and SHAP (SHapley Additive exPlanations) can enhance ML model transparency.
• To improve the process of selecting the most suitable ML model for medical datasets.
• To evaluate the reliability and efficacy of different ML models for heart disease prediction.

Main Methods:

• Utilized data from the England National Health Services Heart Disease Prediction Cohort.
• Compared performance metrics of XGBoost, Random Forest, Artificial Neural Network, and Adaptive Boosting models.
• Employed bootstrap simulation (N=10,000) to derive distributions of model metrics and covariate importance (Gain).
• Applied SHAP to explain ML model outputs and assess the variance in accuracy metrics.

Main Results:

• XGBoost was selected as the optimal model.
• For XGBoost, 10,000 simulations showed AUROC variance (0.176), balanced accuracy variance (0.205), sensitivity variance (0.307), and specificity variance (0.394).
• Covariate gain analysis revealed significant variance for Angina (0.231), Cholesterol (0.178), MaxHR (0.119), and Age (0.098).

Conclusions:

• Empirical evaluation of ML model metric variability through simulation is crucial for transparency and reliability.
• Explanatory algorithms like SHAP help validate covariate importance against existing literature.
• Combining variance statistics with accuracy metrics empowers researchers to select optimal ML models for diverse datasets.