MIDRC-MetricTree: a decision tree-based tool for recommending performance metrics in artificial intelligence-assisted...

Area of Science:

• Medical Imaging
• Machine Learning
• Artificial Intelligence in Healthcare

Background:

• The Medical Imaging and Data Resource Center (MIDRC) was established to support machine learning (ML) research in medical imaging.
• ML algorithms are crucial for tasks like disease detection, diagnosis, and treatment response assessment, particularly in the context of the COVID-19 pandemic.
• A need exists for standardized methods to evaluate the performance of these complex algorithms.

Purpose of the Study:

• To develop a publicly accessible metrology resource for evaluating medical image analysis ML algorithms.
• To provide researchers with tools to assess algorithm performance across various medical imaging tasks.

Main Methods:

• An interactive decision tree, MIDRC-MetricTree, was developed.
• The decision tree is organized by ML task type (classification, detection/localization, segmentation, time-to-event analysis, estimation).
• Users input task details, reference standard nature, and algorithm output type to receive tailored recommendations for performance evaluation.

Main Results:

• MIDRC-MetricTree offers guidance on appropriate performance evaluation approaches and metrics.
• Recommendations include literature references and links to relevant software/code and tutorial videos.
• The decision tree covers diverse scenarios, including classification with binary or multiclass outputs and varying reference standard reliability.

Conclusions:

• The MIDRC-MetricTree is a valuable, publicly available resource for researchers.
• It facilitates task-specific performance evaluations for a wide range of medical imaging ML applications.
• This tool aids in the rigorous assessment of ML algorithms in medical imaging research.