Generation of Training Data to Distinguish Adverse Events from Medical Conditions

To support pharmacovigilance activities in social media, innovative methods are required to detect named entities corresponding to drugs and adverse events. However, annotated resources are missing for French-language discussion forums, and manual annotation to create training datasets is a time-consuming and complex process. We propose an approach based on generative artificial intelligence to detect relevant training examples and annotate 200 user messages from forums based on our annotation guidelines. Two prompting strategies were implemented (zero-shot and few-shot) using a decoder-only large language model (LLM). A comparison was performed between model-generated and manual annotations. Results using partial match evaluation yielded a score of 0.69-0.68 for zero-shot and few-shot prompting. While drug entity detection achieved F1-scores of 0.83, adverse event detection showed limitations (F1: 0.50-0.61), suggesting challenges in disambiguating adverse events from medical conditions in informal text. These findings suggest that while LLMs show promise for drug entity detection and initial filtering of pharmacovigilance relevant messages, the AE-Condition disambiguation task requires further refinement.