Evaluating motivational interview quality using large language models and hidden Markov models.

Area of Science:

• Behavioral Science
• Computational Linguistics
• Artificial Intelligence

Background:

• Motivational Interviewing (MI) is a counseling approach to promote behavior change.
• Traditional MI quality assessment is labor-intensive and subjective.
• An automated framework using LLMs and HMMs is proposed for MI evaluation.

Purpose of the Study:

• Evaluate an LLM-HMM framework for predicting MI session quality.
• Examine motivational state transitions in high- and low-quality MI sessions.

Main Methods:

• Analyzed 40 MI sessions, classifying client utterances with an LLM.
• Used HMMs to model motivational state transitions based on LLM scores.
• Quantified differences in transition matrices and assessed predictive performance using LOOCV.

Main Results:

• High-quality MI sessions showed fluid motivational state transitions; low-quality sessions exhibited persistent resistance.
• A statistically significant difference in transition matrices was found between session quality groups (p < 0.001).
• The LLM-HMM framework achieved 0.80 accuracy in predicting MI session quality via LOOCV.

Conclusions:

• The LLM-HMM framework offers a scalable and objective alternative to manual MI quality assessment.
• Potential future applications include real-time therapist support, training, and prognosis prediction.
• Further validation on field-collected data is recommended.