Evaluating Large Language Models for Sentiment Analysis and Hesitancy Analysis on Vaccine Posts From Social Media:...

Area of Science:

• Computational Linguistics
• Public Health Informatics
• Artificial Intelligence in Healthcare

Background:

• Social media is a key platform for public health discourse, including vaccine discussions.
• Accurate sentiment analysis and hesitancy detection are vital for understanding public opinion.
• Large language models (LLMs) offer advanced capabilities for analyzing complex health-related text data.

Purpose of the Study:

• To evaluate and compare the performance of various LLMs in sentiment analysis and vaccine hesitancy detection.
• To identify the most efficient, accurate, and cost-effective LLM for analyzing vaccine-related public sentiment.
• To assess LLM performance across different learning paradigms (zero-shot, 1-shot, few-shot).

Main Methods:

• Utilized GPT-3.5, GPT-4, Claude-3 Sonnet, and Llama 2 for analyzing vaccine discussions from X, Reddit, and YouTube.
• Tested models across zero-shot, 1-shot, and few-shot learning paradigms.
• Evaluated performance using accuracy, F1-score, precision, and recall, alongside a cost analysis based on token usage.

Main Results:

• GPT-4 achieved the highest performance (F1-score=0.85, accuracy=0.83), outperforming other LLMs.
• Zero-shot learning was found to be as effective as few-shot learning, with lower computational costs.
• Challenges remain in accurately classifying neutral sentiments, sarcasm, and indirect expressions of hesitancy.

Conclusions:

• GPT-4 is the most accurate LLM for vaccine sentiment and hesitancy analysis, with zero-shot learning offering a good balance of performance and efficiency.
• A hybrid approach combining LLMs with traditional machine learning models may optimize cost and performance.
• LLMs show significant potential for public health communication strategies, but ongoing evaluation and refinement are necessary.