Evaluating the Quality of Psychotherapy Conversational Agents: Framework Development and Cross-Sectional Study

Area of Science:

• Artificial Intelligence
• Mental Health Technology
• Digital Therapeutics

Background:

• Growing public use of AI chatbots for simulated psychotherapy presents potential risks.
• A standardized method for evaluating the quality of these AI-driven mental health tools is essential.

Purpose of the Study:

• To introduce the CAPE (Conversational Agent for Psychotherapy Evaluation) framework for assessing psychotherapy chatbot quality.
• To evaluate and compare popular AI psychotherapy chatbots available on the OpenAI GPT store using the CAPE framework.

Main Methods:

• Four popular AI psychotherapy chatbots were identified on the OpenAI GPT store.
• Two independent reviewers applied the CAPE framework, comprising 8 sections with individual quality subscores (0-1), using simulated patient interactions.
• Statistical tests (t tests, Wilcoxon signed rank tests) were used to analyze pairwise differences in subscores.

Main Results:

• Chatbots scored highly on background information, conversational abilities, therapeutic alliance, boundaries, and accessibility.
• Low scores were observed for therapeutic orientation and monitoring/risk evaluation.
• Transparency regarding training data and knowledge base was lacking (subscores=0).
• Significant differences were found in the privacy and harm section, with chatbots inadequately addressing these areas.

Conclusions:

• The CAPE framework provides a reliable method for evaluating psychotherapy chatbot quality, aiding user decision-making.
• Popular AI psychotherapy chatbots demonstrate strengths in rapport and accessibility but exhibit deficiencies in essential safety and privacy functions.