Automating pharmacovigilance evidence generation: using large language models to produce context-aware structured...

Area of Science:

• Pharmacovigilance
• Artificial Intelligence
• Database Management

Background:

• Pharmacovigilance databases require complex queries for safety data retrieval.
• Natural Language Queries (NLQs) are often difficult to translate into Structured Query Language (SQL) for database interaction.
• Large Language Models (LLMs) offer potential for automating NLQ-to-SQL conversion.

Purpose of the Study:

• To enhance information retrieval accuracy in pharmacovigilance databases.
• To develop a method for converting NLQs into SQL queries using LLMs.
• To evaluate the impact of business context on LLM-driven query generation.

Main Methods:

• Utilized OpenAI's GPT-4 model within a retrieval-augmented generation (RAG) framework.
• Enriched the RAG framework with a business context document.
• Assessed LLM performance across varying query complexities (low, medium, high) with and without the business context.

Main Results:

• NLQ-to-SQL accuracy increased from 8.3% (schema only) to 78.3% with the business context document.
• Accuracy improvements were consistent across all query complexity levels.
• Excluding high complexity queries, performance reached up to 85%, indicating strong potential for deployment.

Conclusions:

• Integrating business context significantly improves LLM accuracy for generating executable and semantically correct SQL queries.
• The proposed methodology enhances the accessibility of pharmacovigilance data for non-technical users.
• This approach provides a transferable framework for improving data retrieval in various data-intensive fields.