Need for Objective Task-Based Evaluation of Image Segmentation Algorithms for Quantitative PET: A Study with ACRIN...

Area of Science:

• Medical Imaging
• Radiology
• Computational Biology

Background:

• Accurate segmentation of PET images is vital for clinical applications.
• Current segmentation algorithms are often evaluated using metrics not directly linked to clinical tasks.
• Figures of merit like Dice Similarity Coefficient (DSC), Jaccard Similarity Coefficient (JSC), and Hausdorff Distance (HD) lack clinical task correlation.

Purpose of the Study:

• To investigate the consistency between task-agnostic metrics and clinically relevant quantitative tasks for PET segmentation algorithms.
• To determine if standard metrics accurately reflect performance in estimating metabolic tumor volume (MTV) and total lesion glycolysis (TLG).

Main Methods:

• Retrospective analysis of PET images from the ACRIN 6668/RTOG 0235 trial for non-small cell lung cancer.
• Evaluation of conventional algorithms (thresholding, Snakes, Markov random field-GMM) and a U-net-based deep learning algorithm.
• Comparison of task-agnostic metrics (DSC, JSC, HD) with task-based performance in estimating MTV and TLG.

Main Results:

• Conventional algorithms showed discrepancies: SUVmax40% outperformed SUVmax50% on DSC but had higher bias in MTV/TLG estimation.
• Markov random field-Gaussian mixture model outperformed Snakes on DSC but increased MTV bias.
• For U-net, deeper networks improved MTV/TLG accuracy with reduced bias, despite similar DSC/JSC/HD values.
• Different loss functions for U-net significantly impacted MTV/TLG bias without altering DSC/JSC/HD.

Conclusions:

• Task-agnostic metrics for PET segmentation algorithms may not align with clinical task performance.
• Objective, task-based evaluation is essential for reliable clinical translation of PET segmentation algorithms.
• This study highlights the limitations of standard metrics and the importance of task-specific validation.