Large Language Models Can Enable Inductive Thematic Analysis of a Social Media Corpus in a Single Prompt: Human...

Area of Science:

• Computational linguistics
• Public health informatics
• Social media analytics

Background:

• Manual analysis of social media provides public health insights but is time-intensive.
• Generative large language models (LLMs) offer potential for summarizing and interpreting large text volumes.
• The effectiveness of LLMs in discerning subtle health meanings from social media remains unclear.

Purpose of the Study:

• To assess if LLMs can perform topic model selection and thematic analysis on social media content.
• To compare LLM performance against human subject matter experts from a prior study.
• To determine if LLMs can reasonably identify health-related themes in social media data.

Main Methods:

• Replicated a prior study's research question and social media content using three LLMs (GPT4-32K, Claude-instant-100K, Claude-2-100K).
• Compared LLM topic selection and theme identification with manual human analyses.
• Evaluated LLM consistency and inter-model agreement.

Main Results:

• LLMs ranked previously human-identified topics highly, exceeding chance levels (P<.001).
• LLMs identified relevant themes with low hallucination rates, deemed reasonable by experts.
• Variability in theme identification was observed between different LLMs and repeated analyses.

Conclusions:

• LLMs efficiently process large social media health datasets and extract reasonable themes.
• LLMs show significant potential for automated public health social listening.
• Further validation is needed to match the depth of human expert analysis in theme extraction.