Scaling medical device regulatory science using large language models.

Area of Science:

• Regulatory Science
• Artificial Intelligence
• Medical Device Development

Background:

• The rapid advancement of artificial intelligence (AI) and machine learning (ML) has led to an increase in AI/ML-enabled medical devices.
• Emerging best practices for the development, testing, and monitoring of these devices present regulatory challenges.
• Manual annotation of complex, jargon-filled regulatory documents is time-consuming and limits data analysis.

Purpose of the Study:

• To validate the effectiveness of large language models (LLMs) in analyzing medical device regulatory documents.
• To assess the potential of LLMs to scale data analysis in regulatory science.
• To inform policy-making through up-to-date data analyses of the regulatory landscape.

Main Methods:

• Conducted a wide-ranging validation study of LLMs for analyzing regulatory documents.
• Evaluated LLM outputs using expert manual annotations.
• Employed an "LLM-as-a-judge" approach for performance assessment.

Main Results:

• LLMs demonstrated high accuracy, often exceeding 80%, in extracting attributes from regulatory documents.
• LLMs successfully extracted information relevant to both pre-market and post-market settings.
• The study confirmed LLMs' capability to scale data analysis for regulatory science applications.

Conclusions:

• LLMs offer a powerful and accurate solution for overcoming the challenges of analyzing unstructured regulatory data.
• LLM-driven analysis can significantly enhance the monitoring of device validation, medical device reporting, and identification of adverse event risks.
• This research highlights the transformative potential of LLMs in advancing medical device regulatory science and policy.