Unregulated large language models produce medical device-like output

Area of Science:

• Artificial Intelligence in Medicine
• Clinical Informatics
• Medical Device Regulation

Background:

• Large language models (LLMs) demonstrate significant potential for enhancing clinical decision support (CDS) systems.
• Currently, no LLM-based tools have received authorization from the Food and Drug Administration (FDA) for use as CDS devices.
• The integration of advanced AI into healthcare necessitates a thorough evaluation of its capabilities and regulatory implications.

Purpose of the Study:

• To assess the feasibility of inducing large language models (LLMs) to generate outputs suitable for clinical decision support (CDS).
• To determine if current LLMs can produce device-like CDS output across various clinical scenarios.
• To inform regulatory considerations for the potential deployment of LLMs in clinical practice.

Main Methods:

• Evaluation of two prominent large language models (LLMs).
• Testing LLM performance in generating clinical decision support (CDS) outputs.
• Analysis of LLM responses across a diverse range of simulated clinical scenarios.

Main Results:

• Large language models (LLMs) demonstrated the ability to produce outputs resembling device-like clinical decision support (CDS).
• Consistent decision support generation was observed across multiple tested clinical scenarios.
• The study indicates that LLMs can readily mimic the functionality of approved CDS devices.

Conclusions:

• The findings suggest that large language models (LLMs) are capable of providing device-like clinical decision support (CDS).
• The ease with which LLMs produce CDS-like output highlights a potential gap in current regulatory frameworks.
• Formal deployment of LLMs for clinical use necessitates the development of appropriate regulatory oversight and authorization processes by bodies like the FDA.