Exploring the Efficacy of Large Language Models in Summarizing Mental Health Counseling Sessions: Benchmark Study

Area of Science:

• Natural Language Processing in Mental Health
• Artificial Intelligence for Clinical Support
• Computational Linguistics in Psychotherapy

Background:

• Manual summarization of mental health counseling sessions is time-consuming.
• Efficient session summaries are crucial for continuity of care and therapy planning.
• Existing automatic summarization methods often fail to capture counseling nuances.

Purpose of the Study:

• To evaluate state-of-the-art large language models (LLMs) for aspect-based summarization of therapy sessions.
• To benchmark LLM performance in selectively summarizing distinct counseling components.
• To assess the utility of AI-driven summarization for mental health professionals.

Main Methods:

• Development of a novel dataset: Mental Health Counseling-Component-Guided Dialogue Summaries (191 sessions).
• Assessment of 11 LLMs on aspect-based summarization of counseling dialogues.
• Quantitative evaluation using ROUGE and BERTScore, complemented by qualitative expert review.

Main Results:

• Task-specific LLMs (MentalLlama, Mistral, MentalBART) outperformed others on quantitative metrics (ROUGE, BERTScore).
• Expert review indicated Mistral surpassed MentalLlama and MentalBART in 6 key parameters.
• All models showed limitations in summarizing opportunity costs and perceived effectiveness.

Conclusions:

• LLMs fine-tuned on mental health data achieve higher automatic evaluation scores.
• Expert assessments reveal current LLMs are not yet reliable for direct clinical application.
• Further research and validation are essential for practical implementation in mental healthcare.