Beyond Simulations: What 20,000 Real Conversations Reveal About Mental Health AI Safety.

Area of Science:

• Artificial Intelligence
• Mental Health Technology
• Computational Linguistics

Background:

• Large language models (LLMs) are increasingly used in mental health applications.
• Current safety evaluations for LLMs often use limited, simulation-based benchmarks.
• Real-world language data is crucial for accurate AI safety assessment.

Purpose of the Study:

• To evaluate the safety of a purpose-built mental health AI compared to general-purpose LLMs.
• To assess the effectiveness of layered safeguards in real-world user interactions.
• To determine the utility of ecological audits for AI safety estimation.

Main Methods:

• Replication of four published safety evaluations for general-purpose and mental health LLMs.
• Ecological audit of 20,000 real user conversations with a purpose-built mental health AI.
• Clinician review of conversations to identify suicide risk and non-suicidal self-injury (NSSI) handling.

Main Results:

• The purpose-built AI generated significantly less harmful content across suicide/NSSI, eating disorder, and substance use benchmarks compared to general LLMs.
• In real user data, zero suicide-risk cases were identified without crisis resources.
• A low false negative rate of 0.38% was estimated for NSSI mentions lacking intervention.

Conclusions:

• Purpose-built AI with layered safeguards demonstrates improved safety for mental health applications.
• Ecological audits provide a valuable method for assessing AI safety in real-world scenarios.
• Findings support the development and deployment of specialized AI for mental health support.