Retrieval augmented large language model system for comprehensive drug contraindications

Area of Science:

• Medical Informatics
• Artificial Intelligence in Healthcare
• Pharmacovigilance

Background:

• Large language models (LLMs) show versatility but face challenges in healthcare, especially for critical data like pharmaceutical contraindications.
• Accurate and reliable information on drug contraindications is essential for patient safety and effective healthcare.
• Existing LLM applications require enhancement to reliably handle complex medical information, such as drug-drug interactions and patient-specific warnings.

Purpose of the Study:

• To enhance the capability of LLMs in identifying pharmaceutical contraindications accurately.
• To implement and evaluate a Retrieval Augmented Generation (RAG) pipeline for improved contraindication detection.
• To reduce uncertainty in prescription and drug intake decisions through precise contraindication information.

Main Methods:

• Utilized OpenAI's GPT-4o-mini as the base LLM and text-embedding-3-small for embeddings.
• Integrated a hybrid retrieval system with re-ranking using the LangChain framework.
• Leveraged Drug Utilization Review (DUR) data focusing on age, pregnancy, and concomitant drug use contraindications.

Main Results:

• Baseline LLM accuracy for contraindications ranged from 0.49 to 0.57.
• The RAG pipeline significantly improved accuracy to 0.94 (age), 0.87 (pregnancy), and 0.89 (concomitant use).
• Demonstrated substantial reduction in uncertainty for prescription and drug intake decisions.

Conclusions:

• Augmenting LLMs with a RAG framework substantially improves the accuracy of pharmaceutical contraindication information.
• The developed RAG pipeline offers a promising solution for reliable drug safety information retrieval.
• This approach can enhance clinical decision-making and patient safety in medication management.