Benchmarking Motivational Interviewing Competence of Large Language Models

Area of Science:

• Artificial Intelligence in Healthcare
• Clinical Psychology
• Natural Language Processing

Background:

• Motivational interviewing (MI) is crucial for behavioral change in substance use disorders.
• The Motivational Interviewing Treatment Integrity (MITI) framework assesses MI fidelity.
• Large language models (LLMs) show potential for generating MI-consistent therapist responses, but their performance in clinical settings is under-researched.

Purpose of the Study:

• To benchmark the MI competence of proprietary and open-source LLMs using the MITI framework on real-world clinical transcripts.
• To compare LLM performance against human therapists.
• To assess the distinguishability of LLM-generated responses from human therapists' responses.

Main Methods:

• Evaluated 10 LLMs (3 proprietary, 7 open-source) using the MITI 4.2 framework on 96 handcrafted and 34 real-world clinical transcripts.
• Generated parallel LLM-therapist utterances, keeping client responses static.
• Conducted a distinguishability experiment with two psychiatrists to differentiate LLM from human responses.

Main Results:

• All tested LLMs exhibited fair to good MI competence (MITI global scores >3.5).
• Top LLMs outperformed human experts in Complex Reflection percentage (96% vs 39%) and Reflection-Question ratio (>2.8 vs 1.2).
• Psychiatrists identified LLM responses with only 56% accuracy, indicating subtle differences.

Conclusions:

• LLMs achieve good MI proficiency in real-world clinical transcripts according to the MITI framework.
• High complex reflection rates in LLMs may lead to unnatural dialogue.
• Open-source LLMs are promising for expanding MI counseling in underserved areas, pending further clinical validation.