Governing autonomous AI in clinical care: the window is narrowing

Area of Science:

• Medical Informatics
• Health Policy
• Clinical AI Governance

Background:

• Artificial intelligence (AI) systems are increasingly used in clinical settings for autonomous actions like scheduling and prior authorization.
• Primary care faces challenges with physician workload, making it a prime area for AI deployment.
• Current governance structures for AI in healthcare are insufficient, lacking accountability and monitoring.

Purpose of the Study:

• To highlight the gap between AI capabilities in healthcare and the lagging governance infrastructure.
• To emphasize the need for improved oversight to ensure safe and equitable AI deployment.
• To propose a framework for governing autonomous AI actions in clinical care.

Main Methods:

• Analysis of the current state of AI in clinical care and its governance.
• Identification of risks associated with autonomous AI actions.
• Review of existing healthcare resource inequities.

Main Results:

• AI systems are moving towards autonomous actions, outpacing governance development.
• Accountability mechanisms, performance standards, and post-deployment monitoring for AI are weak.
• Existing resource inequities can be mirrored or exacerbated by inadequate AI oversight.

Conclusions:

• Health systems and policymakers must risk-stratify autonomous AI actions based on clinical error consequences.
• Procurement and payment models should be linked to minimum AI evaluation standards.
• Shared governance infrastructure with sustainable funding is essential for equitable AI integration.