Transformer-based models for ADR detection: cross-drug validation and benchmarking against large language models

Area of Science:

• Pharmacovigilance and computational linguistics.
• Application of machine learning in drug safety monitoring.

Background:

• Adverse drug reactions (ADRs) are critical safety concerns.
• Social media offers valuable real-time patient data but requires advanced processing.
• Natural language processing (NLP) and transfer learning are key to analyzing unstructured text.

Purpose of the Study:

• To assess transformer models fine-tuned for ADR classification on tweets about GLP-1 receptor agonists.
• To benchmark these models against state-of-the-art large language models (LLMs).

Main Methods:

• Fine-tuning BERT-base, BERTweet-base, and GPT-2 models using ADR datasets.
• Testing models on 396 tweets mentioning GLP-1 receptor agonists.
• Evaluating performance using F1 scores and comparing against ChatGPT 4o, ChatGPT 4o-mini, and Gemini 2.5 Flash.

Main Results:

• BERTweet-base achieved the highest F1 score (0.729) among fine-tuned transformer models.
• LLMs demonstrated superior performance, with ChatGPT 4o-mini achieving an F1 score of 0.948.
• LLMs substantially outperformed fine-tuned transformer models in ADR classification.

Conclusions:

• Fine-tuned transformer models show potential for ADR detection in social media.
• State-of-the-art LLMs, especially ChatGPT 4o-mini, offer significantly improved capabilities for pharmacovigilance.
• LLMs represent a powerful advancement for real-time drug safety surveillance.