ChatGPT as a mental health advisory service: Comparing evaluations from youth and health professionals

Despite the growing use of artificial intelligence in youth mental health support, little is known about how either young people or health professionals perceive answers to mental health-related inquiries generated by large language models (LLMs). Therefore, we draw on Media Richness Theory to examine how these two user groups perceive the "richness" of text-based communication in this context and whether young people and health professionals differ in their assessment.

A total of 123 young people and 31 health professionals evaluated answers to youth mental health inquiries. Each inquiry had two blinded answers: one generated by ChatGPT (GPT-4) and one written by a health professional. Participants rated the answers for validation, relevance, clarity, and utility and were asked to recommend one or both answers. Open-ended responses elaborating participant choices were also collected.

The quantitative findings show that young people and health professionals rated answers from both sources similarly on validation, clarity, and utility. However, young people rated ChatGPT's answers higher for relevance and utility, finding them "richer." This was supported by qualitative data, where youth preferred ChatGPT's clear and actionable answers. Health professionals showed no strong preference and were more critical, often finding the answers too detailed or lacking empathy.

This study is the first to compare youth and professional perspectives on ChatGPT's role in youth mental health advice within a blind evaluation design. We conclude by proposing a hybrid advisory model that combines professional expertise with LLMs to enhance the efficiency, scale, and accessibility of youth mental health advisory services.