Using Large Language Models to Detect and Understand Drug Discontinuation Events in Web-Based Forums: Development and...

Area of Science:

• Natural Language Processing
• Health Informatics
• Computational Linguistics

Background:

• Large language models (LLMs) like BART and GPT-4 are transforming unstructured text analysis, including healthcare applications.
• Analyzing social media data offers public health insights, but detecting drug discontinuation events (DDEs) remains a challenge.
• Identifying DDEs is critical for understanding medication adherence and patient outcomes.

Purpose of the Study:

• To develop a flexible framework for clinical research in data-sparse environments.
• To identify DDEs and their root causes using LLMs in the MedHelp web forum.
• To release the first open-source DDE datasets to facilitate future research.

Main Methods:

• Utilized LLMs (GPT-4 Turbo, GPT-4o, DeBERTa, BART) for DDE detection and root cause analysis in MedHelp user comments.
• Employed zero-shot classification for model predictions without task-specific training.
• Classified user comments into sentences and applied various strategies to evaluate model performance.

Main Results:

• GPT-4o achieved the highest accuracy in determining DDE root causes with a 12.9% incorrect prediction rate (hamming loss).
• BART excelled in DDE detection among open-source models, yielding an F1-score of 0.86 without fine-tuning.
• The dataset contained 10.7% DDEs, demonstrating model robustness in imbalanced data.

Conclusions:

• Open- and closed-source LLMs (GPT-4o, BART) effectively detect DDEs and root causes via zero-shot classification from public data.
• The proposed framework is robust and scalable for addressing data-sparse clinical research questions.
• The release of open-access DDE datasets is expected to drive further research and discovery in pharmacovigilance.