Automated Safety Testing and Reporting Application for Conversational Safety Monitoring of Generative AI Tools for...

Area of Science:

• Digital Health
• Artificial Intelligence in Mental Health
• Clinical Informatics

Background:

• AI-driven conversational tools are expanding in mental health care, enhancing access and scalability.
• These AI systems present safety risks due to user disclosures (e.g., self-harm ideation) and inadequate AI responses.

Purpose of the Study:

• To develop and evaluate the Automated Safety Testing and Reporting Application (ASTRA).
• ASTRA aims to detect clinically relevant risk behaviors in AI-mediated mental health conversations.

Main Methods:

• ASTRA was tested on 100 synthetic therapeutic conversations created by clinicians.
• Conversations included diverse risk behaviors and AI responses, with human coder consensus as the benchmark.
• Performance was assessed using diagnostic metrics and agreement statistics across two prompt iterations.

Main Results:

• ASTRA demonstrated high concordance with expert human ratings across all risk categories (accuracy >0.90).
• Specificity was uniformly high; sensitivity ranged from 0.55-1.00.
• Agreement beyond chance was substantial to almost perfect (κ=0.65-1.00), notably accurate for subtle self-harm indicators.

Conclusions:

• ASTRA reliably identified mental health-related risk behaviors at the conversation level in an initial validation.
• Findings support the feasibility of independent AI safety monitoring systems in mental health.
• Further evaluation with larger, real-world datasets is recommended.