When are large language models medical devices? Navigating the MDR and AI Act for healthcare applications

The flexibility of generative artificial intelligence (AI), and specifically large language models (LLMs), holds promise to transform healthcare by offering new tools to improve patient care and enhance clinician efficiency. However, this flexibility introduces new challenges, as LLM applications might inadvertently be used in ways that do not align with the intended purpose of the manufacturer, raising patient safety concerns and liability issues. We have identified a regulatory gap whereby LLM applications that could be repurposed, malfunction or be inadvertently misused for medical purposes fall outside the scope of the Medical Device Regulation (MDR) because they lack a medical intended purpose, and outside meaningful AI Act safety obligations because they do not qualify as high-risk systems, yet still present patient safety risks without adequate regulatory guidance. In this manuscript, we aim to address this regulatory gap for LLM applications in healthcare by identifying these ambiguous grey-area applications and providing four recommendations-validation, mitigation, compliance and alignment-focused on prevention and risk mitigation rather than reclassification as medical devices. These recommendations are intended to advance the responsible implementation of LLM applications in healthcare while ensuring patient safety.