Monitoring Methodology for an AI Tool for Breast Cancer Screening Deployed in Clinical Centers

Area of Science:

• Medical Imaging and Artificial Intelligence
• Radiology and Clinical Informatics

Background:

• Artificial intelligence (AI) tools are increasingly used in medical diagnostics, necessitating robust monitoring systems.
• Ensuring the consistent performance of AI in breast cancer screening across different clinical settings is crucial for patient safety and diagnostic accuracy.

Purpose of the Study:

• To propose and evaluate a methodology for monitoring the performance of an AI tool deployed for breast cancer screening in clinical centers.
• To assess the real-world behavior of an AI mammography tool by comparing its performance metrics against a reference standard.

Main Methods:

• An AI tool for detecting suspicious regions in mammograms and assigning suspicion scores was deployed in four US radiological centers.
• Data from 36,581 AI records were collected between April 2021 and December 2022.
• AI performance was monitored by comparing its score distribution in each center against an in silico reference distribution using the Pearson correlation coefficient (PCC).

Main Results:

• The AI demonstrated consistent performance across all four centers, with PCC values ranging from 0.975 to 0.998.
• The high PCC values indicate that the AI tool behaved as expected in the clinical deployment.
• The proposed monitoring methodology successfully captured the AI's performance consistency.

Conclusions:

• The developed methodology provides a reliable way to monitor AI tools for breast cancer screening in clinical practice.
• Low PCC values can serve as alerts for potential software malfunctions, enhancing the safety and reliability of deployed AI.
• This approach can lead to the development of new indicators for improved monitoring of hospital software.