Quantifying the clinical relevance of a laboratory observer performance paradigm

Area of Science:

• Medical imaging analysis
• Radiology observer performance
• Quantitative medical imaging

Background:

• Laboratory observer performance measurements, such as receiver operating characteristic (ROC) and free-response ROC (FROC), may not accurately represent actual clinical interpretations.
• These discrepancies can potentially compromise the clinical relevance of laboratory-based performance assessments.

Purpose of the Study:

• To develop a novel method for quantifying the clinical relevance of laboratory performance paradigms.
• To compare the clinical relevance of ROC and FROC paradigms in a nodule detection task.

Main Methods:

• Prospective interpretations of 80 digital chest radiographs were evaluated against a truth panel.
• Receiver operating characteristic (ROC) and free-response ROC (FROC) data were collected from 21 radiologists.
• A pseudovalue-based image-level correctness measure was computed, and its association with clinical correctness was assessed using a relevance ROC curve (rAUC).

Main Results:

• Low correlations and near chance-level rAUC values were observed between clinical and laboratory paradigms, highlighting discrepancies.
• The area under the relevance ROC curve (rAUC) demonstrated greater sensitivity to differences in clinical relevance compared to traditional measures.
• Confidence intervals for interparadigm differences were narrower for rAUC compared to conventional measures.

Conclusions:

• The developed relevance ROC (rAUC) method provides a more sensitive measure of clinical relevance for laboratory paradigms.
• A new rAUC method is proposed for quantifying the clinical relevance of laboratory observer performance measurements.
• This approach can help bridge the gap between laboratory findings and real-world clinical application in medical imaging.