Should LLMs be over empowered for high-stake regulatory research?

Area of Science:

• Pharmacology
• Artificial Intelligence
• Regulatory Science

Background:

• Recent studies explored zero-shot and few-shot learning for regulatory tasks using large language models (LLMs).
• Models like Flan-T5 demonstrated effectiveness in extracting drug-drug interactions and intrinsic factors from FDA drug labels.
• However, significant implementation challenges were identified.

Purpose of the Study:

• To critically evaluate the feasibility of using open-source large language models (LLMs) in regulatory research.
• To discuss potential mitigation strategies for identified LLM implementation challenges.

Main Methods:

• Critical evaluation of existing research on LLMs in regulatory tasks.
• Analysis of challenges such as computational constraints, performance variability, prompt sensitivity, and misclassification risk.
• Discussion of intuitive methods to address these limitations.

Main Results:

• Open-source LLMs, like Flan-T5, can achieve high precision in extracting specific data (e.g., pharmacokinetic drug-drug interactions) from regulatory documents.
• Key challenges include computational demands, inconsistent performance, sensitivity to input prompts, and potential for errors.

Conclusions:

• Implementing open-source LLMs in regulatory research is feasible but requires careful consideration of limitations.
• Developing intuitive strategies is crucial for overcoming computational, performance, and accuracy challenges associated with LLM deployment in this domain.