Identifying Dietary Supplements Related Effects from Social Media by ChatGPT

Area of Science:

• Pharmacovigilance
• Natural Language Processing
• Social Media Analytics

Background:

• Identifying drug-supplement interactions is crucial for public health.
• Social media platforms offer vast, real-time data for pharmacovigilance.
• Existing datasets for drug-supplement interactions are limited.

Purpose of the Study:

• To expand the drug-supplement interactions dataset (iDisk) by analyzing dietary supplement-related tweets.
• To advance relationship identification methodologies in social media data.
• To compare the performance of traditional NLP models with large language models for this task.

Main Methods:

• Collected over 90,000 tweets from 2007-2022.
• Annotated 1,000 tweets for entities and relationships.
• Employed a BioBERT model and ChatGPT-generated prompts for entity and relationship identification.

Main Results:

• BioBERT model achieved an F1 score of 0.90 for relationship prediction.
• ChatGPT prompts reached an F1 score of 0.99 for relationship prediction.
• Entity type recognition faced challenges due to high semantic similarity between types.

Conclusions:

• The study significantly enhances relationship identification from social media, especially for dietary supplement usage.
• Combining traditional NLP with large language models shows promise for complex healthcare text analysis.
• The methodology offers improved post-market surveillance and public health monitoring capabilities.