Health Systems Govern Only the Tip of the AI Iceberg

AbstractPhysician-facing generative AI is now ubiquitous in medicine, with over two thirds of physicians using AI daily in their practice. With many physicians dissatisfied with their health system's AI adoption speed, it is not surprising that most use occurs outside institutional oversight. AI governance within health systems is generally designed to assess AI tools oriented around specific use cases, which allows systems to quantify the potential harms of such tools so that they can be mitigated. This approach to governance fails for general-purpose generative AI platforms because the range of harms and benefits varies depending on how physicians use them. The result is that direct oversight of narrowly focused AI tools coexists with widespread under-the-surface use of consumer AI platforms with uncertain data privacy and security practices. Health systems' lack of engagement with consumer AI tools worsens their risk by eliminating institutional visibility and protections. We describe the scope of this problem and why health systems need to reform their approach to AI governance.