Alignment of Large Language Model Responses With Human Therapists in Motivational Interviewing.

Area of Science:

• Artificial Intelligence in Mental Health
• Computational Linguistics
• Psychotherapy Research

Background:

• Large language models (LLMs) are increasingly explored for mental health applications.
• Evaluating the alignment of LLM responses with evidence-based psychotherapy is crucial.
• Motivational interviewing (MI) offers a structured framework for such evaluations.

Purpose of the Study:

• To assess the alignment of LLM-generated responses with human therapist responses in MI sessions.
• To utilize automated similarity metrics for quantitative comparison.
• To investigate the impact of topic consistency on response alignment.

Main Methods:

• Cross-sectional study using high-fidelity MI transcripts from public counseling videos.
• GPT-4o LLM generated responses based on MI-informed prompts and conversational context.
• Alignment assessed via cosine similarity (semantic overlap) and DeepEval (contextual appropriateness).
• Therapist topic-consistency index analyzed as a moderator.

Main Results:

• 3706 therapist turns from 154 MI sessions were analyzed.
• DeepEval scores (0.72) were higher than cosine similarity scores (0.29), indicating better contextual appropriateness than semantic overlap.
• Therapist topic consistency significantly moderated both metrics.
• LLM performance slightly declined in longer conversations, showing reduced grounding.

Conclusions:

• Prompted LLMs demonstrate general alignment with therapist responses in MI.
• Limitations in long-range coherence and stylistic alignment necessitate improvements.
• Clinical validation and refined evaluation methods are required before LLM integration into mental health care.