Reconceiving Safety Regulation for AI and ML Medical Software

This article explores the challenges of regulating AI and ML clinical decision support tools intended to assist trained health care professionals in delivering clinical care. Two old, twentieth-century regulatory models have dominated discussions of medical AI policy since 2013. Thinking inside these old regulatory boxes has not produced effective regulatory solutions to address the novel risks AI poses in clinical care. The first regulatory box treats software as a medical device, which tasks medical device regulators with making software safe but neglects the crucial roles physicians, nurses, administrators, medical practice regulators, and other health oversight bodies must also play to make AI-enabled health care safe for patients. The second box views AI-enabled healthcare as a complex sociotechnical system where the central regulatory challenge is to incentivize the creation of "slack" at the human-AI interface: that is, to inject buffers, redundancies, and checks and balances that enhance opportunities for human actors to intervene if the software runs amok. The 21st Century Cures Act of 2016 favored this latter model, but it has not been successfully implemented. This article then offers two more conceptualizations. Viewed through the lens of the old corporate practice of medicine doctrine, the central oversight challenge in AI-enabled health care is to prevent non-physician corporate actors (for example, software developers) from overriding or corrupting physicians' ability to exercise independent medical judgment on behalf of their patients. The final conceptualization likens AI and ML tools to intelligent agents that are "colonizing" the health care system and threatening various harms to the indigenous humans-patients, health professionals, and administrators-who inhabit it. Protecting the safety, culture, and values of health care may require institutional and regulatory reforms that go far beyond merely repurposing old regulatory frameworks left over from the past century.