Human-in-the-loop reconsidered: Shadow use and reliance management in drug development

This article examines the ethical governance of artificial intelligence (AI) use in drug development through joint principles of good AI practice issued by the U.S. Food and Drug Administration (FDA) and the European Medicines Agency (EMA). It argues that the significance of the principles lies in moving beyond AI exceptionalism: AI should neither be uniformly prohibited nor uniformly permitted but assessed in a risk-based manner according to context, purpose, and potential impact across the drug lifecycle. Among the ethical and governance risks associated with AI, this study focuses on two organizational risks that are particularly relevant to implementation. The first is shadow use, in which AI involvement remains insufficiently visible, documented, or reviewed. The second is reliance management. Once AI is integrated into research and regulatory workflows, some degree of reliance is inevitable; however, such reliance must remain conscious, proportionate, reviewable, and supported by meaningful human oversight. Overreliance and deskilling are risks associated with poorly managed reliance. Ethical governance should therefore make AI use visible and reviewable while preserving the practical ability to question, verify, escalate, or set aside AI-assisted outputs.