Automating inductive thematic analyses of health content using large language models: a proof-of-concept study using...

Area of Science:

• Computational linguistics
• Qualitative research methods
• Social media analysis

Background:

• Inductive thematic analysis requires significant domain expertise.
• Large language models (LLMs) face challenges in replicating this expert-driven process.
• Automating thematic analysis of social media data is a key research goal.

Purpose of the Study:

• To evaluate the feasibility of using LLMs for expert-driven thematic analysis of social media data.
• To assess LLM performance in replicating human coding of qualitative data.
• To explore the potential of LLMs as a scalable supplement for qualitative research.

Main Methods:

• Utilized two Reddit datasets on xylazine for model optimization and validation.
• Evaluated five LLMs against expert coding using binary classifications.
• Employed zero-, single-, and few-shot prompting strategies, measuring performance via accuracy, precision, recall, and F1 score.

Main Results:

• GPT-4o with 2-shot prompting achieved 90.9% accuracy and a 0.71 F1 score on the validation set.
• Model-derived thematic distributions closely matched expert classifications for high-prevalence themes.
• LLM performance indicated a strong potential for replicating expert qualitative analysis.

Conclusions:

• Few-shot LLM-based approaches can effectively automate thematic analysis.
• LLMs offer a scalable and efficient supplement to traditional qualitative research methods.
• This study demonstrates the practical application of LLMs in analyzing social media discourse.