Walking Backward to Ensure Risk Management of Large Language Models in Medicine

Area of Science:

• Medical Device Regulation
• Artificial Intelligence in Healthcare
• Risk Management

Background:

• Specialized Large Language Models (LLMs) trained on medical data are increasingly used as AI medical devices.
• Current risk management under the EU Medical Device Regulation (MDR) employs a "forward-walking" approach.
• This approach may be insufficient for addressing the unique risks of LLM-based medical devices.

Purpose of the Study:

• To analyze how providers of specialized LLMs manage risks under the EU MDR.
• To evaluate the suitability of the current "forward-walking" risk management approach for AI medical devices.
• To propose an improved risk management framework for LLM-based medical devices.

Main Methods:

• Review of the EU MDR risk management process.
• Case study analysis using an AI medical device for lung cancer detection.
• Conceptualization of a "backward-walking" risk management logic.

Main Results:

• The "forward-walking" approach in MDR is misaligned with the intended use requirements for AI medical devices.
• This approach leads to incomplete risk assessment, overlooking new hazards and use cases.
• It fails to adequately address the specific risks inherent in specialized LLMs.

Conclusions:

• The EU MDR risk management framework requires a "backward-walking" logic for AI medical devices.
• This backward-reasoning approach enables better identification of intended users, hazards, and use cases.
• It facilitates a more robust risk management strategy for LLM-based medical devices, ensuring safety and efficacy.