S-MEOD: A Novel Evaluation Metric for Frame-Based Medical Object Detection

Area of Science:

• Medical Imaging Analysis
• Computer Vision
• Artificial Intelligence in Healthcare

Background:

• Traditional object detection metrics like precision, recall, mAP, and F-score are insufficient for frame-based medical data.
• These metrics can misrepresent model effectiveness by not accounting for temporal consistency, leading to misleading performance evaluations.
• Existing metrics fail to capture clinically relevant detection behavior in sequential medical imaging.

Purpose of the Study:

• To introduce a novel evaluation metric, S-MEOD (Sequential Method of Evaluation for Object Detection), for frame-based medical object detection.
• To address the limitations of traditional metrics in assessing object detection models in sequential medical imaging.
• To provide a more comprehensive and clinically relevant performance assessment for medical AI.

Main Methods:

• Developed S-MEOD, combining Sequence-aware Precision (SaP) and Sequence-oriented Detection (SoD).
• Evaluated S-MEOD on frame-based sequences using object detection models, including YOLO architectures.
• Conducted experiments on coronary angiography data to compare S-MEOD with traditional metrics.

Main Results:

• S-MEOD demonstrated a more accurate and intuitive reflection of model effectiveness in frame-based detection compared to traditional metrics.
• On coronary angiography data, increasing confidence thresholds improved precision and mAP but drastically reduced recall and F1-score.
• S-MEOD showed a deterioration in temporal detection performance as thresholds increased, aligning better with clinical relevance.

Conclusions:

• S-MEOD accurately reflects clinically relevant detection behavior, distinguishing sparse detections from sequence-level failures.
• The proposed S-MEOD metric offers an easy-to-interpret and reliable alternative for evaluating frame-based medical object detection models.
• Adoption of S-MEOD can improve clinical applicability assessment and redefine performance standards in medical imaging research.