ESR Essentials: common performance metrics in AI-practice recommendations by the European Society of Medical Imaging...

Area of Science:

• Medical Imaging
• Artificial Intelligence in Healthcare
• Radiology Informatics

Background:

• Artificial intelligence (AI) is increasingly integrated into radiology workflows.
• Evaluating the performance of AI tools is crucial for ensuring patient safety and clinical utility.
• Radiologists require standardized methods for assessing AI performance.

Purpose of the Study:

• To provide radiologists with practical recommendations for evaluating AI performance in radiology.
• To outline key performance metrics for AI in medical imaging.
• To address common pitfalls and offer mitigation strategies for AI evaluation.

Main Methods:

• Review of key performance metrics including segmentation overlap, test-based metrics (sensitivity, specificity, AUC-ROC), and outcome-based metrics (precision, NPV, F1-score, MCC, AUC-PR).
• Emphasis on local validation using independent datasets and consideration of deployment context.
• Discussion of common pitfalls like metric overreliance, misinterpretation in low-prevalence settings, and workflow integration challenges.

Main Results:

• Key recommendations include task-specific metric selection, local validation, and considering the clinical context.
• Strategies are provided to mitigate pitfalls such as threshold selection and prevalence-adjusted evaluation.
• Guidance on assessing AI-generated image quality is also included.

Conclusions:

• Radiologists must critically evaluate AI performance metrics, recognizing their limitations and the need for clinical utility assessment.
• Independent, context-specific evaluation is essential for safe and effective AI deployment in radiology.
• Aligning performance metrics with the AI's intended task (segmentation, detection, classification) and target population is vital for improving diagnostic accuracy and patient outcomes.