Public Health Discussions on Social Media: Evaluating Automated Sentiment Analysis Methods

Area of Science:

• Computational Social Science
• Natural Language Processing
• Public Health Informatics

Background:

• Sentiment analysis is crucial for social media research, but choosing between manual and automated methods requires guidance.
• Opioid epidemic discussions on YouTube present a complex, unbalanced dataset for sentiment analysis.

Purpose of the Study:

• To compare the performance of popular Natural Language Processing (NLP) tools (VADER, TEXT2DATA, LIWC-22) and a large language model (ChatGPT 4.0) against manual sentiment coding.
• To evaluate sentiment analysis methods based on practical considerations like ease of programming and cost.

Main Methods:

• Manual sentiment coding using an inductive, iterative content analysis approach.
• Quantitative evaluation using descriptive statistics, ROC curve analysis, confusion matrices, and agreement metrics (Cohen κ, accuracy, precision, sensitivity, F1-score, MCC).

Main Results:

• LIWC-22's tone summary measure excelled at estimating negative sentiment prevalence in the unbalanced dataset.
• VADER demonstrated the best performance in classifying manually coded negative comments via ROC analysis.
• NLP tools showed fair agreement with manual coding (Cohen κ), while ChatGPT 4.0 exhibited poor agreement and failed in multiple attempts.

Conclusions:

• VADER is recommended as a cost-free tool for sentiment analysis, especially for longer comments, due to its strong discrimination capabilities.
• LIWC-22's tone summary is advised for estimating negative comment prevalence in unbalanced datasets.
• ChatGPT 4.0 has not yet surpassed NLP models for analyzing highly unbalanced social media data.