Developing large language models to detect adverse drug events in posts on x

Area of Science:

• Pharmacovigilance and Drug Safety
• Natural Language Processing
• Computational Linguistics

Background:

• Adverse drug events (ADEs) are a significant cause of hospitalizations, morbidity, and mortality.
• Post-marketing surveillance is crucial for drug safety, traditionally relying on systems like FAERS.
• Social media offers rich, unstructured patient data for enhanced drug safety research.

Purpose of the Study:

• To develop and evaluate large language models (LLMs) for automated adverse drug event (ADE) classification in social media data.
• To compare the performance of various fine-tuned LLMs and ChatGPT prompting strategies for ADE detection.
• To identify key linguistic features indicative of ADEs in social media text.

Main Methods:

• Fine-tuning of several LLMs including BERT-base, Bio_ClinicalBERT, RoBERTa, and RoBERTa-large on X (formerly Twitter) data.
• Experimentation with ChatGPT few-shot prompting and a fine-tuned ChatGPT model.
• Comprehensive model evaluation using metrics such as sensitivity, specificity, PPV, NPV, accuracy, F1-measure, and AUC.

Main Results:

• RoBERTa-large achieved the highest F1-measure of 0.8, outperforming other evaluated models.
• A fine-tuned ChatGPT model achieved a respectable F1-measure of 0.75.
• Feature importance analysis identified terms like 'withdrawals', 'dry', 'mouth', and 'paralysis' as significant indicators of ADEs.

Conclusions:

• LLMs show significant potential for augmenting ADE detection in post-marketing drug safety surveillance.
• RoBERTa-large and fine-tuned ChatGPT models offer promising performance for analyzing social media data for ADEs.
• The identified clinically relevant features underscore the value of LLMs in understanding drug safety signals from unstructured text.