Exploring large language models for summarizing and interpreting an online brain tumor support forum

Area of Science:

• Natural Language Processing
• Health Informatics
• Computational Linguistics

Background:

• Qualitative data analysis in health research is time-consuming.
• Large Language Models (LLMs) offer potential for automating text summarization.
• Evaluating LLM performance against traditional methods is crucial for adoption.

Purpose of the Study:

• To compare the summarization capabilities of GPT-3.5, GPT-4, and Llama 3 for qualitative health data.
• To assess LLM-generated summaries against traditional thematic analysis.
• To evaluate the efficiency and consistency of LLMs in health research contexts.

Main Methods:

• Collected qualitative data from an online brain tumor support forum.
• Analyzed data using traditional thematic coding (Dedoose) and LLM summarization.
• Employed metrics like ROUGE, METEOR, and BERTScore for evaluation.
• Calculated readability scores (Flesch-Kincaid) for all summaries.

Main Results:

• GPT-4 outperformed GPT-3.5 and Llama 3 on ROUGE and METEOR metrics.
• LLM summaries closely matched human-generated thematic analysis.
• GPT-4 processed entire transcripts, enhancing efficiency over segmented models.
• Significant reductions in time and labor were observed with LLM use.

Conclusions:

• LLMs, especially GPT-4, show strong potential for summarizing complex qualitative health data efficiently.
• These tools can enhance research efficiency and support patient-centered care.
• Further investigation into LLM biases and limitations is warranted.