A deployment safety case for AI-assisted prostate cancer diagnosis

Area of Science:

• Medical Informatics
• Artificial Intelligence in Healthcare
• Clinical Safety Engineering

Background:

• Deep learning (DL) systems offer clinical benefits but require more than regulatory approval for safe real-world use.
• Post-deployment hazardous events can emerge unpredictably from complex healthcare settings and system interactions.
• Hospitals often mandate self-verification for diagnostic devices, highlighting the need for ongoing safety assurance.

Purpose of the Study:

• To address the critical need for continual safety assurance of regulatorily approved DL systems in clinical deployment.
• To present a systematic methodology for identifying and mitigating deployment-related hazards for AI medical devices.
• To establish a framework for the ongoing monitoring of safety for AI systems post-approval.

Main Methods:

• Mapping the clinical workflow of the deployed DL system (Paige for prostate cancer diagnosis).
• Conducting hazard and risk analysis tailored to the specific clinical workflow.
• Developing a deployment safety case to support initial deployment and continuous monitoring.

Main Results:

• A systematic approach was developed to identify potential hazards arising from the deployment of an FDA-approved DL system.
• The methodology provides a structured basis for ensuring the ongoing safety and reliability of AI diagnostic tools in clinical practice.
• The study demonstrates the importance of proactive safety assessment beyond initial regulatory approval.

Conclusions:

• Regulatory approval is a necessary but not sufficient condition for the safe clinical deployment of deep learning systems.
• A robust safety case and continuous monitoring framework are essential to manage emergent risks associated with AI in healthcare.
• This work offers a practical methodology for assuring the safety of AI-driven diagnostic tools throughout their lifecycle.