Framework for Testing Robustness of Machine Learning-Based Classifiers.

Area of Science:

• Biomedical Informatics
• Computational Biology
• Artificial Intelligence in Medicine

Background:

• Artificial intelligence (AI) and machine learning (ML) are increasingly used for diagnostic biomarker classifiers.
• Assessing the robustness and uncertainty of these AI/ML models is crucial but underexplored.

Purpose of the Study:

• To propose and validate a framework for evaluating the robustness of existing AI/ML-based diagnostic classifiers.
• To assess classifier performance variability, parameter changes, and feature importance under data perturbations.

Main Methods:

• Utilized factor analysis and Monte Carlo simulations to evaluate classifier robustness.
• Assessed input feature importance and output/parameter variability in response to data perturbations.
• Applied the framework to six common AI/ML techniques on a metabolomics dataset.

Main Results:

• The framework successfully predicted the relative robustness of different classifiers without recomputation.
• Quantified classifier performance variability and identified key input features.
• Demonstrated the ability to estimate a priori a classifier's tolerance to noise while maintaining accuracy.

Conclusions:

• The proposed framework provides a reliable method for assessing the robustness of AI/ML diagnostic classifiers.
• This approach aids in understanding biomarker reliability and selecting robust models for clinical applications.
• The findings highlight the importance of robustness evaluation in AI/ML biomarker development.