Assessing Mental Health Counseling Interventions with LLMs

Large Language Models (LLMs) are becoming increasingly valuable for psychologists analyzing the large volume of data that is collected from human subjects transcripts, talk-based therapy inventions, and other types of counseling sessions. This paper presents a case study on using LLMs to assess the quality of counseling interventions. The aim of this system was to evaluate the treatment fidelity of school clinicians delivering a group-based behavioral parent training intervention for ADHD (Attention Deficit Hyperactivity Disorder). We examine the following research questions: 1) how consistent is the LLM in its assessment if queried repeatedly? 2) how well do LLMs agree with expert human coders? 3) do different LLMs agree with one other? Our results show that Gemini slightly outperforms GPT-4o at assessing treatment fidelity and produced qualitatively superior explanations for its decisions. Based on our experiments, we propose a set of best practices for using LLMs for evaluating counseling interventions.