Harnessing large language models' zero-shot and few-shot learning capabilities for regulatory research

Area of Science:

• Artificial Intelligence
• Natural Language Processing
• Computational Biology

Background:

• Large language models (LLMs) offer advanced conversational capabilities but often require data transmission to external servers.
• Online LLM use poses data privacy risks, especially for sensitive information.
• Organizations prioritizing data protection, like regulatory agencies, need secure, local AI solutions.

Purpose of the Study:

• To evaluate the feasibility of implementing open-source LLMs within a secure local network.
• To assess LLM performance in extracting clinical pharmacology information from drug labels.
• To determine the efficacy of LLMs for sensitive data processing in regulated environments.

Main Methods:

• Implementation of various open-source LLMs within a regulatory agency's local network.
• Performance assessment using few-shot and zero-shot learning on specific NLP tasks.
• Evaluation of a selected LLM for identifying drug exposure factors without fine-tuning.

Main Results:

• Some open-source LLMs achieved performance comparable or superior to traditional models with minimal training.
• A selected LLM accurately identified factors affecting drug exposure with 78.5% accuracy on a large dataset.
• The study demonstrated successful local deployment for sensitive data analysis.

Conclusions:

• Open-source LLMs can be effectively implemented in secure local networks for sensitive data tasks.
• LLMs offer a viable solution for natural language processing when extensive training data is unavailable.
• This approach enhances data privacy and security for regulatory and other high-priority organizations.