Feasibility-First Risk Management for Clinical AI: A Deterministic Systems Intelligence Framework for Release,...

Area of Science:

• Medical Informatics
• Artificial Intelligence in Healthcare
• Clinical Risk Management

Background:

• Artificial intelligence (AI) is widely adopted in healthcare for tasks like prediction, diagnosis, and patient monitoring.
• Current safety discussions focus on model performance, bias, and explainability, but overlook the critical question of when AI outputs should be integrated into clinical practice.
• A gap exists in managing the transition of AI outputs from technical plausibility to actionable clinical use.

Purpose of the Study:

• To propose a Deterministic Systems Intelligence (DSI) framework for feasibility-first clinical AI risk management.
• To establish an admissible-state layer for evaluating AI outputs before clinical or organizational action.
• To define governance states for AI output release, ensuring responsible integration into healthcare regimes.

Main Methods:

• Development of the Deterministic Systems Intelligence (DSI) framework.
• Introduction of an admissible-state layer to assess AI outputs based on data fitness, population fit, clinical actionability, workflow capacity, equity, authority, monitoring, and reversibility.
• Classification of AI outputs into five governance states: release, restricted release, active monitoring, rollback, and HOLD.

Main Results:

• The DSI framework provides a structured approach to AI risk management by evaluating release readiness.
• The admissible-state layer ensures AI outputs meet predefined criteria before clinical action, classifying them into distinct governance states.
• A case study demonstrated how the framework manages AI deterioration alerts based on local context and safety controls, including the option to HOLD outputs when insufficient.

Conclusions:

• Responsible clinical AI necessitates more than just predictive capabilities; it requires release readiness, robust monitoring, and the capacity to withhold AI outputs.
• The DSI framework complements existing safety and regulatory approaches by adding a crucial admissibility step before AI action.
• Implementing the DSI framework enhances the safe and effective integration of AI into healthcare settings by prioritizing feasibility and risk management.