Assessing reader performance in radiology, an imperfect science: lessons from breast screening

Area of Science:

• Radiology
• Medical Imaging
• Public Health

Background:

• Mammography screening programs rely on accurate reader performance for effective cancer detection.
• Clinical audit and standardized test sets are primary methods for assessing reader accuracy.
• Existing assessment methods present limitations that may affect the reliability of reader performance evaluation.

Purpose of the Study:

• To review the limitations of current methods for assessing reader accuracy in mammography screening.
• To critically evaluate the metrics used in clinical audits and standardized test sets.
• To identify factors impacting the external validity of screening test sets.

Main Methods:

• Literature review of current mammography reader assessment techniques.
• Analysis of metrics used in clinical audits (e.g., recall rate).
• Examination of standardized screening test sets and their associated experimental confounders.

Main Results:

• Clinical audit metrics like recall rate may not accurately reflect specificity.
• Standardized test sets, while convenient, suffer from experimental confounders affecting clinical relevance.
• Four key factors (scrutiny, environment artificiality, response simplification, abnormality prevalence) impact test set external validity.

Conclusions:

• Current methods for assessing mammography reader accuracy have significant limitations.
• The clinical relevance of standardized test sets is questionable due to artificiality and confounders.
• Further research is needed to understand the correlation between clinical audit performance and test set performance.