Semantic Search of FDA Guidance Documents Using Generative AI

Area of Science:

• Regulatory Science
• Artificial Intelligence in Medicine

Background:

• Generative artificial intelligence (AI) offers transformative potential for accessing information crucial to regulating human drug and biologic products.
• Efficient information retrieval is vital for regulatory processes.

Purpose of the Study:

• To evaluate a generative AI application with retrieval-augmented generation (RAG) architecture for accurately answering questions based on FDA guidance documents.
• To determine the efficacy of different large language models (LLMs) within a RAG framework for regulatory information retrieval.

Main Methods:

• Five LLMs (Flan-UL2, GPT-3.5 Turbo, GPT-4 Turbo, Granite, Llama 2) were tested with the Golden Retriever RAG application.
• Models were configured for precise answers (low temperature) to ensure reliable regulatory guidance.

Main Results:

• GPT-4 Turbo demonstrated the highest performance among the evaluated LLMs.
• GPT-4 Turbo provided correct responses with additional helpful information 33.9% of the time and fully correct responses 35.7% of the time.
• The RAG application successfully cited the correct source document in 89.2% of cases.

Conclusions:

• Generative AI applications can expedite the retrieval of information from FDA guidance documents.
• The risk of incorrect information necessitates further refinement of AI models before widespread adoption in critical drug development decisions.
• Prompt engineering and parameter tuning may enhance the accuracy and completeness of AI-generated responses.