Machine learning models' assessment: trust and performance.

Area of Science:

• Medical Informatics
• Machine Learning in Healthcare
• Clinical Decision Support

Background:

• Machine learning (ML) models are often black boxes, hindering trust and adoption in healthcare.
• Lack of trust is a significant barrier to the widespread application of ML in clinical settings.
• A simultaneous evaluation of trust and performance is needed for reliable healthcare ML tools.

Purpose of the Study:

• To develop and validate an evaluation framework assessing both trust and performance of ML models.
• To compare the trust and performance of various ML models against a clinical standard for cardiovascular risk stratification.
• To identify ML models suitable for clinical application based on trust and performance metrics.

Main Methods:

• Trust assessment incorporated model robustness, confidence intervals (95% CI), and interpretability via feature ranking comparison with clinical evidence.
• Performance was evaluated using the geometric mean.
• Five models (GRACE score, logistic regression, Naïve Bayes, decision trees, rule-based approach) were compared using a Portuguese cardiovascular risk dataset (N=1544).

Main Results:

• Simultaneous assessment of trust and performance was successfully implemented.
• The rule-based approach exhibited a high level of operational trust.
• The rule-based approach outperformed the GRACE score in performance and enhanced physician acceptance.

Conclusions:

• The developed evaluation approach effectively assesses ML model trust and performance concurrently.
• The rule-based approach shows significant potential for clinical application in cardiovascular risk assessment.
• Improved trust and performance of ML models can enhance physician acceptance and aid clinical decision-making.