Intelligent Framework for Adverse Drug Event Identification Using Large Language Models and Retrieval-Augmented...

Area of Science:

• Natural Language Processing
• Artificial Intelligence in Medicine
• Pharmacovigilance

Background:

• Adverse drug events (ADEs) present significant public health and economic challenges.
• Extracting ADE information from unstructured clinical notes is difficult due to semantic complexity.
• Large language models (LLMs) show promise for text comprehension but can suffer from domain-specific hallucinations.

Purpose of the Study:

• To evaluate the effectiveness of retrieval-augmented generation (RAG) for identifying ADEs in Chinese clinical narratives using LLMs.
• To establish a paradigm for ADE identification in clinical text using RAG and LLMs.

Main Methods:

• A dataset of 18,432 high-quality Chinese clinical notes was curated and annotated.
• A gold-standard reference dataset (n=2510) and an ADE knowledge base (n=5144) were established.
• Three LLMs (DeepSeek-V3, ERNIE 3.5-8K, GPT-4o) were evaluated using nonaugmented generation (NAG), static-augmented generation (SAG), and RAG strategies, with performance assessed via precision, recall, and F1-score at multiple recognition levels.

Main Results:

• The first Chinese ADE corpus from clinical notes was created and released.
• RAG consistently outperformed NAG and SAG, achieving an optimal L3 F1-score of 0.9638 with DeepSeek-V3.
• RAG significantly improved GPT-4o's recall from 0.6419 to 0.9241 and demonstrated clinical utility with high specificity (0.9821) on real-world data.

Conclusions:

• Integrating a domain-specific knowledge base with LLMs via RAG is effective for accurate ADE identification in Chinese clinical notes.
• This RAG strategy mitigates LLM hallucinations, providing a benchmark for pharmacovigilance and drug safety research.
• The developed framework supports clinical decision support systems and advances drug safety monitoring.