Summarizing Online Patient Conversations Using Generative Language Models: Experimental and Comparative Study.

Area of Science:

• Natural Language Processing
• Artificial Intelligence in Healthcare
• Patient-Reported Outcomes

Background:

• Social media provides valuable patient experience data for regulatory bodies.
• Current methods for analyzing patient data are either manual and not scalable or provide only quantitative insights.
• There is a need for methods that can automatically summarize patient texts and yield qualitative insights at scale.

Purpose of the Study:

• To evaluate the effectiveness of state-of-the-art large language models in summarizing patient posts from online health communities.
• To compare the performance of different language models and prompting strategies for summarizing individual patient experiences.

Main Methods:

• Applied three language models (Flan-T5, GPT-3, and GPT-3.5) with various prompting strategies to summarize patient posts.
• Evaluated generated summaries against 124 manually created reference summaries.
• Utilized Recall-Oriented Understudy for Gisting Evaluation (ROUGE) and BERTScore metrics for comparison.

Main Results:

• GPT-3.5 demonstrated superior performance compared to other models using zero-shot prompting, based on ROUGE and BERTScore.
• The best results were achieved with GPT-3.5 using directional stimulus prompting in a 3-shot setting.
• Manual review indicated that the summaries generated by the best-performing method were accurate and plausible.

Conclusions:

• Pretrained language models offer a valuable tool for extracting qualitative patient experience insights, identifying unmet needs and priorities.
• These insights can inform healthcare delivery improvements and patient-centered drug development.
• Limitations include a small data sample and a single annotator for reference summaries; results may not generalize to all models and strategies.