Risk-based evaluation of machine learning-based classification methods used for medical devices.

Area of Science:

• Medical device evaluation
• Machine learning in healthcare
• Risk assessment methodologies

Background:

• Machine learning (ML) is increasingly used in medical devices.
• Risk and cost considerations are crucial for evaluating medical device performance.
• This study focuses on risk-based evaluation for ML-based classification models.

Purpose of the Study:

• To assess the current utilization of risk-based metrics in ML-based classification models.
• To introduce an approach integrating risks and costs into ML model performance metrics.
• To analyze the impact of risk ratios on overall performance and regulatory compliance.

Main Methods:

• Literature research on risk-based metrics in ML medical device publications.
• Development of an approach for risk- and cost-integrated performance evaluation.
• Analysis of risk ratio impacts and alignment with EU medical device regulations.

Main Results:

• Most current publications lack risk-based performance metrics for ML models.
• Risk considerations significantly impact evaluation outcomes, with potential risk increases up to 196%.
• Risk-based approaches are necessary for ML medical device assessment under EU regulations.

Conclusions:

• A risk-based approach is essential for evaluating ML-based medical devices.
• Current scientific literature often neglects risk considerations in ML model assessment.
• The proposed methodology aligns with EU regulatory requirements for medical devices.