Generation of Training Data to Distinguish Adverse Events from Medical Conditions

Area of Science:

• Natural Language Processing
• Pharmacovigilance
• Artificial Intelligence

Background:

• Pharmacovigilance requires robust methods for detecting drug and adverse event (AE) entities in social media.
• Annotated French-language datasets for social media pharmacovigilance are lacking.
• Manual annotation is resource-intensive and time-consuming.

Purpose of the Study:

• To explore generative artificial intelligence (AI) for creating annotated datasets for French social media pharmacovigilance.
• To evaluate the performance of large language models (LLMs) in detecting drug and AE entities.
• To compare model-generated annotations with manual annotations.

Main Methods:

• Utilized a decoder-only LLM with zero-shot and few-shot prompting strategies.
• Annotated 200 French user messages from discussion forums.
• Compared LLM-generated annotations against manually created annotations using partial match evaluation.

Main Results:

• LLMs demonstrated promise in detecting drug entities with an F1-score of 0.83.
• Adverse event detection showed limitations, with F1-scores ranging from 0.50 to 0.61.
• Partial match evaluation scores were 0.69 for zero-shot and 0.68 for few-shot prompting.

Conclusions:

• Generative AI, specifically LLMs, can assist in drug entity detection and initial filtering of pharmacovigilance-relevant messages from social media.
• Disambiguating adverse events from medical conditions in informal text remains a challenge for current LLM approaches.
• Further refinement is needed for LLM-based AE-Condition disambiguation in pharmacovigilance.