Assessing the accuracy of automatic speech recognition for psychotherapy

Area of Science:

• Mental Health
• Computational Linguistics
• Clinical Psychology

Background:

• Accurate transcription of psychotherapy sessions is crucial for improving therapy effectiveness, clinician training, and safety monitoring.
• Commercial automatic speech recognition (ASR) software exists, but its performance and suitability in mental health settings are not well-established.
• Defining appropriate metrics and accuracy thresholds for ASR in diverse clinical applications, from population analysis to individual safety, remains unclear.

Purpose of the Study:

• To evaluate the feasibility and accuracy of ASR in psychotherapy settings.
• To determine the suitability of current ASR technology for various clinical use cases in mental health.
• To assess the performance of ASR for specific clinical indicators like depression and harm-related content.

Main Methods:

• Development and implementation of a HIPAA-compliant ASR system for psychotherapy recordings.
• Quantitative analysis of transcription accuracy using word error rate (WER).
• Calculation of sensitivity and positive predictive value for depression-related and harm-related utterances.

Main Results:

• The ASR system achieved a 25% word error rate for general psychotherapy transcription.
• For depression-related utterances, ASR demonstrated 80% sensitivity and 83% positive predictive value.
• Clinician-identified harm-related sentences had a higher ASR word error rate of 34%.

Conclusions:

• ASR is technically feasible in psychotherapy but requires further accuracy enhancements for broad clinical adoption.
• Current ASR performance may support research on language patterns and treatment variations in depression.
• The demonstrated word error rates suggest ASR is not yet sufficiently reliable for real-time, individual-level safety surveillance in psychotherapy.