A novel statistical framework for quantifying risks and benefits of AI automation in screening mammography

Area of Science:

• Radiology
• Artificial Intelligence
• Medical Imaging

Background:

• Artificial intelligence (AI) is explored for triaging mammograms to reduce radiologist workload.
• Optimal AI
• rule-out
• threshold determination remains unclear.

Purpose of the Study:

• To present a framework for determining an optimal AI
• rule-out
• threshold in mammography screening.
• To analyze the trade-offs between caseload reduction and false omission rates at different AI thresholds.

Main Methods:

• Retrospective analysis of 114,229 bilateral 2D digital screening mammograms (2006-2023).
• AI scoring using Mirai (open-source deep-learning model) for 1-year risk.
• Evaluation of metrics: Caseload Reduction Rate (CRR), Gross AI False Omission Rate (G-FOR), AI Net False Omission Rate (N-FOR), and AI Adjusted Net False Omission Rate (AN-FOR[30%]).
• Comparison of two thresholds: 0.20 (Youden's J) and 0.05 (AN-FOR[30%]=0).

Main Results:

• At the 0.20 threshold (Youden's J): CRR=75%, G-FOR=0.26%, N-FOR=0.17%, AN-FOR[30%]=0.14% (121 missed cancers).
• At the 0.05 threshold (AN-FOR[30%]=0): CRR=36%, G-FOR=0.12%, N-FOR=0.07%, AN-FOR[30%]=0.00% (0 missed cancers).
• The Youden's J threshold yields a 75% caseload reduction with 121 additional missed cancers.

Conclusions:

• Radiology practices can utilize the proposed framework to select appropriate AI
• rule-out
• thresholds based on specific clinical goals.
• A threshold of 0.05 (AN-FOR[30%]=0) achieves zero additional missed cancers at the cost of a 36% caseload reduction.
• A threshold of 0.20 (Youden's J) maximizes caseload reduction (75%) but results in 121 additional missed cancers.