Development, validation and AI chatbot evaluation of the Adolescent Mental Health AI Response Tool

Many adolescents seek mental health advice from general-purpose artificial intelligence (AI) chatbots, despite limited evaluations against youth-relevant clinical standards. We developed the Adolescent Mental Health AI Response Tool (AMHART), a clinician-informed instrument for auditing the presence of guideline-informed response content across seven domains, and applied it to four major AI chatbots. Candidate items were reviewed by clinicians for their evidence base, objectivity, and clinical importance, yielding a subset with strong content validity and inter-rater reliability. We then configured a three-model large language model evaluator ensemble, assessed its agreement with human reference labels, and scored multi-turn conversations in which adolescent personas (varying by sex, diagnosis, and symptom severity) engaged with ChatGPT-5, ChatGPT-4, Claude-4.5, and Gemini-2.5. AMHART scores indicated greater coverage of applicable criteria for anxiety than depression and for moderate than severe presentations. Findings suggest AMHART offers a reliable, clinician-informed benchmark for evaluating chatbot responses to adolescent anxiety and depression disclosures.