When Is It Safe to Introduce an AI System Into Healthcare? A Practical Decision Algorithm for the Ethical...

Area of Science:

• Medical Informatics
• Artificial Intelligence in Medicine
• Clinical Decision Support

Background:

• Growing global interest in AI tools for healthcare applications.
• Concerns exist regarding the risks of AI in healthcare, particularly opaque 'black box' systems, and their impact on patient informed consent.
• Interpretability of AI models is challenging for advanced AI like generative AI and large language models (LLMs).

Purpose of the Study:

• To propose a shift in evaluating clinical AI tools from interpretability to implementation risk.
• To introduce a practical decision algorithm for assessing and managing the risks associated with implementing 'black box' AI in clinical settings.
• To ensure patient safety and informed consent when using AI in healthcare.

Main Methods:

• Development of a decision algorithm to evaluate the implementation risk of AI tools.
• Assessment of AI implementation risk based on technical robustness, implementation feasibility, and analysis of harms and benefits.
• Categorization of AI systems into minimal-risk, moderate-risk, or high-risk based on the evaluation.
• Consideration of cost-effectiveness and patient informed consent within the algorithm.

Main Results:

• The proposed algorithm provides a framework for the clinical implementation of 'black box' AI.
• AI systems are categorized into minimal-risk (standard use), moderate-risk (innovative use), and high-risk (experimental use).
• Implementation recommendations are risk-proportional, with higher-risk categories requiring increased oversight.

Conclusions:

• Evaluating AI for clinical use should prioritize implementation risk over interpretability, especially for LLMs and generative AI.
• The decision algorithm offers a practical approach to managing risks associated with AI implementation in healthcare.
• The framework supports safe and effective integration of AI tools while upholding patients' informed consent and considering cost-effectiveness.