Defining operational safety in clinical artificial intelligence systems

Area of Science:

• Medical Artificial Intelligence
• Clinical Decision Support Systems
• Health Informatics

Background:

• Current artificial intelligence (AI) adoption in clinical settings primarily emphasizes automation.
• Traditional accuracy metrics do not adequately address the critical question of AI system trustworthiness and operational safety.
• Evaluating AI safety requires metrics beyond mere statistical accuracy.

Purpose of the Study:

• Introduce the Safety-Aware Receiver Operating Characteristic (SA-ROC) framework to define and assess operational safety for AI systems.
• Establish clear criteria for when it is safe to trust AI in clinical workflows.
• Develop a quantitative measure for the workload associated with AI indecision.

Main Methods:

• Developed the Safety-Aware Receiver Operating Characteristic (SA-ROC) framework, defining operational safety based on pre-specified reliability levels.
• Delineated the SA-ROC curve into Rule-in Safe Zone, Rule-out Safe Zone (autonomous AI action), and Gray Zone (mandated human review).
• Introduced the Gray Zone Area (ΓArea) metric to quantify the operational cost of AI indecision and non-automated workload.

Main Results:

• The SA-ROC framework demonstrated a key reversal in a case study of two FDA-cleared cancer screening algorithms.
• The AI model with a statistically superior Area Under the Curve (AUC) was found to be operationally less safe for high-confidence screening tasks.
• The Gray Zone Area (ΓArea) metric quantified the workload implications of AI indecision.

Conclusions:

• The SA-ROC framework enables active governance of AI in healthcare by translating clinical policy into optimized operational workflows.
• This approach complements existing regulatory safety evaluations by focusing on operational safety and trustworthiness.
• SA-ROC provides a crucial tool for ensuring the safe and effective clinical adoption of artificial intelligence.