A Comprehensive Review of Performance Metrics for Computer-Aided Detection Systems

Area of Science:

• Medical Imaging Analysis
• Radiology Informatics
• Computational Pathology

Background:

• Computer-aided detection (CAD) systems are crucial for identifying lung nodules in computed tomography (CT) images.
• Accurate performance evaluation of CAD systems is essential for clinical adoption and improvement.
• Existing evaluation metrics have limitations that can affect the interpretation of CAD system performance.

Purpose of the Study:

• To provide a structured analysis of performance metrics for CAD systems in lung nodule detection.
• To examine the strengths and limitations of various evaluation metrics.
• To offer guidelines for selecting appropriate metrics for CAD system evaluation.

Main Methods:

• Categorization of evaluation methods into per-scan and per-nodule approaches.
• Analysis of key metrics including Area Under the Receiver Operating Characteristic (ROC) curve (AUROC).
• Discussion of nodule-level sensitivity, Free-Response ROC (FROC) curves, Competition Performance Metric (CPM), and Alternative FROC (AFROC).

Main Results:

• AUROC is a key metric for per-scan analysis, assessing the distinction between scans with and without nodules.
• Per-nodule analysis often uses nodule-level sensitivity at fixed false positives per scan, FROC, and CPM.
• CPM has limitations due to its unnormalized scores and data dependency; AFROC was introduced to combine per-scan and per-nodule strengths.

Conclusions:

• Understanding the principles and relative strengths of different metrics is vital for accurate CAD system evaluation.
• The choice of metric impacts the interpretation of clinical utility and practical application of lung nodule detection CAD systems.
• AFROC offers a potential improvement by integrating benefits of both per-scan and per-nodule evaluation frameworks.