Performance of Algorithms Submitted in the 2023 RSNA Screening Mammography Breast Cancer Detection AI Challenge

Area of Science:

• Radiology and Medical Imaging
• Artificial Intelligence in Healthcare
• Oncology

Background:

• The 2023 RSNA Screening Mammography Breast Cancer Detection AI Challenge aimed to develop AI for independent mammogram interpretation.
• Numerous AI algorithms were submitted to assess their capability in detecting breast cancer from screening mammograms.

Purpose of the Study:

• To evaluate the performance of submitted AI algorithms for mammogram interpretation.
• To explore the benefits of combining top-performing AI algorithms into ensemble models.
• To investigate the impact of patient demographics and clinical factors on AI performance.

Main Methods:

• 1537 AI algorithms were evaluated on a dataset from the US and Australia.
• Cancer detection was confirmed via pathological examination; non-cancer cases were followed for at least one year.
• Ensemble models were created by combining top algorithms, and performance was analyzed across different demographic and clinical subgroups.

Main Results:

• The median sensitivity across all AI algorithms was 27.6%, with the top algorithm achieving 48.6%.
• Ensemble models of the top 3 and top 10 algorithms demonstrated significantly higher sensitivity (60.7% and 67.8%, respectively) while maintaining low recall rates.
• AI performance varied by location, with lower sensitivity on US data compared to Australian data, and higher sensitivity for invasive cancers versus non-invasive cancers.

Conclusions:

• Individual AI algorithms identify distinct breast cancers on screening mammograms.
• Ensemble models significantly enhance AI sensitivity for breast cancer detection.
• AI performance is influenced by cancer type and patient population characteristics, necessitating further research for equitable application.