Using Large Language Models to Support Content Analysis: A Case Study of ChatGPT for Adverse Event Detection

Area of Science:

• Artificial Intelligence
• Biomedical Research
• Computational Linguistics

Background:

• Content analysis of social media data is crucial for identifying adverse events (AEs).
• Manual annotation by humans is time-consuming and resource-intensive.
• Large language models (LLMs) offer potential for automating and improving content analysis efficiency.

Purpose of the Study:

• To evaluate the efficacy of ChatGPT, a large language model, in identifying adverse events (AEs) from social media data.
• To compare ChatGPT's performance against human annotators in detecting AEs related to delta-8-tetrahydrocannabinol.
• To assess the potential of LLMs in enhancing the efficiency of biomedical research content analysis.

Main Methods:

• A case study was conducted using social media posts concerning delta-8-tetrahydrocannabinol.
• ChatGPT was provided with identical instructions given to human annotators for AE detection.
• Performance was measured by comparing ChatGPT's AE detection rates with those of human annotators, using Fleiss' kappa for inter-rater reliability.

Main Results:

• ChatGPT demonstrated high agreement with human annotators in detecting any adverse event (94.4%, κ=0.95).
• ChatGPT achieved near-perfect agreement in identifying serious adverse events (99.3%, κ=0.96).
• The results indicate ChatGPT's capability to accurately and efficiently replicate human annotation tasks.

Conclusions:

• Large language models, exemplified by ChatGPT, show significant potential for accurate and efficient content analysis in biomedical research.
• LLMs can effectively assist in identifying adverse events from unstructured data like social media.
• Further research is recommended to explore generalizability across different models, datasets, and analytical tasks.