Binary and multi-category ratings in a laboratory observer performance study: a comparison | JoVE Visualize

Area of Science:

Radiology and Medical Imaging
Diagnostic Performance Evaluation
Observer Performance Studies

Background:

Assessing radiologist performance in screening mammography interpretation is crucial for early cancer detection.
Traditional binary (recall/no recall) and semi-continuous rating scales are used, but their comparative impact on performance is debated.

Purpose of the Study:

To compare radiologist performance using a binary BI-RADS rating scale versus a semi-continuous receiver operating characteristic (ROC) type rating scale in retrospective mammogram interpretation.
To evaluate the consistency of performance metrics between these two rating scales.

Main Methods:

Nine experienced radiologists retrospectively interpreted 155 mammographic examinations using both a binary (recall/not recall) BI-RADS scale and a semi-continuous (0-100) ROC-type scale.
The vertical distance between each reader's binary operating point and their empirical ROC curve was calculated to assess performance differences.
Statistical analysis, including confidence intervals and p-values, was used to determine the significance of observed differences.

Main Results:

On average, radiologists demonstrated similar performance regardless of whether they used a binary or semi-continuous rating scale.
Four readers performed better with the semi-continuous scale, while four performed better with the binary scale; one reader's performance was identical on both.
Only one reader showed a statistically significant difference between their binary operating point and their empirical ROC curve.

Conclusions:

The choice between a binary or semi-continuous rating scale does not systematically alter radiologist performance in retrospective observer studies of screening mammography.
Results suggest that both rating scales yield consistent performance outcomes, supporting the reliability of retrospective observer performance studies.
The findings indicate that retrospective studies using either rating method can provide comparable insights into radiologist performance in mammography interpretation.