Topic-Aware Summarization of Lived Health Care Experiences: Large Language Model Evaluation Study.

Area of Science:

• Health Services Research
• Natural Language Processing
• Qualitative Health Research

Background:

• Traditional analysis of patient feedback relies on surveys and social media, often yielding coarse insights.
• Existing research primarily focuses on patient experiences, neglecting caregivers and healthcare providers.
• Storytelling offers a rich, underexplored avenue for understanding healthcare disparities and improvements.

Purpose of the Study:

• To extract fine-grained insights from diverse healthcare narratives using large language models (LLMs).
• To apply topic detection and hierarchical summarization to long-form individual stories.
• To evaluate LLM-generated summaries using an LLM-as-a-judge framework and expert validation.

Main Methods:

• Utilized 50 transcribed African American health narratives.
• Applied Latent Dirichlet Allocation (LDA) for topic identification.
• Employed an LLM-based hierarchical summarization approach, with GPT-4 for quality assessment and expert validation.

Main Results:

• Identified 26 relevant topics in African American health experiences, including health behaviors and caregiving.
• LLM-generated topic summaries demonstrated high accuracy, comprehensiveness, and usefulness, with minimal fabrication.
• GPT-4 ratings showed moderate-to-high agreement with expert assessments.

Conclusions:

• Latent Dirichlet Allocation (LDA) and LLMs can effectively identify and summarize diverse health experiences.
• This approach offers promising avenues for health research and clinical improvements for patients and caregivers.
• Leveraging narrative data with LLMs can enhance understanding of health disparities and inform better healthcare delivery.