Advanced topic modeling with large language models: analyzing social media content from dementia caregivers

Area of Science:

• Natural Language Processing
• Computational Social Science
• Artificial Intelligence

Background:

• Traditional topic modeling struggles with semantic understanding in social media data.
• Dementia caregiver online discussions present unique challenges for topic analysis.
• Large language models (LLMs) offer advanced semantic comprehension capabilities.

Purpose of the Study:

• To explore the direct application of LLMs for topic modeling of dementia caregiver tweets.
• To leverage LLMs' semantic understanding for more coherent topic generation.
• To evaluate LLM performance against traditional and state-of-the-art topic modeling methods.

Main Methods:

• A dataset of 231,870 dementia caregiver tweets was analyzed using ChatGPT.
• A novel 2-stage batching approach was developed to handle context length limitations.
• LLM performance was compared against 11 baseline methods, including LDA, GSDMM, and BERTopic.
• Topic quality was assessed using Sentence-BERT coherence scores and expert evaluation.

Main Results:

• The LLM-based approach achieved a significantly higher coherence score (0.358) than all baseline methods.
• Traditional methods (GSDMM, LDA) and BERTopic variants showed lower coherence scores.
• The 2-stage batching strategy effectively managed the large dataset while maintaining topic quality.
• Expert evaluation confirmed the relevance and comprehensiveness of LLM-generated topics.

Conclusions:

• LLMs provide a superior methodology for large-scale topic modeling of social media data.
• LLMs demonstrate enhanced ability to capture semantic relationships, leading to more coherent topics.
• This approach offers interpretable insights into caregiver experiences, informing targeted support strategies.