Aligning Large Language Models for Enhancing Psychiatric Interviews Through Symptom Delineation and Summarization:...

Area of Science:

• Artificial Intelligence in Mental Health
• Natural Language Processing in Psychiatry
• Computational Psychiatry

Background:

• Large language models (LLMs) are advancing rapidly across many fields.
• Psychiatric interviews, though structured, remain an under-explored area for LLM applications.
• This study analyzes counseling data from North Korean defectors with trauma and mental health concerns.

Purpose of the Study:

• To determine if LLMs can identify psychiatric symptoms and their indicators within conversation transcripts.
• To assess LLM capabilities in summarizing patient stressors and symptoms from interview dialogues.

Main Methods:

• LLMs were tasked with extracting stressors, delineating symptoms with indicative transcript sections, and summarizing patient information.
• Mental health experts provided labeled data for training and evaluating LLM performance.
• Both zero-shot inference with GPT-4 Turbo and fine-tuning methods were employed.

Main Results:

• LLMs achieved over 0.8 accuracy in symptom delineation using fine-tuning, outperforming zero-shot GPT-4 Turbo.
• Summaries generated by LLMs demonstrated high coherence (4.66) and relevance (4.67) via G-Eval.
• Retrieval-augmented generation did not significantly enhance LLM performance in this context.

Conclusions:

• LLMs, through expert-labeled data and appropriate prompting, can achieve high accuracy in psychiatric symptom delineation.
• LLMs show significant potential in assisting mental health practitioners by enhancing psychiatric interviews.
• This research highlights the effectiveness of LLMs in analyzing complex mental health data.