Predicting Satisfaction With Chat-Counseling at a 24/7 Chat Hotline for the Youth: Natural Language Processing Study

Area of Science:

• Mental Health Technology
• Natural Language Processing
• Youth Support Services

Background:

• Chat-based counseling offers accessible mental health support for young people.
• Natural Language Processing (NLP) is well-suited for analyzing chat interactions to improve care.
• Automated evaluation of user satisfaction is crucial for quality control in these services.

Purpose of the Study:

• To evaluate the feasibility of using NLP for automated assessment of user satisfaction in chat-based mental health support.
• To develop and compare machine learning models for predicting chat helpfulness based on conversation content.
• To inform quality improvement strategies for online youth mental health services.

Main Methods:

• Trained and evaluated machine learning classifiers on approximately 140,000 messages from 2609 young users.
• Utilized an extreme gradient boosting (XGBoost) classifier with a word vectorizer and compared it with transformer-based models.
• Selected the best-performing model for final evaluation on a separate test set of 522 users.

Main Results:

• The XGBoost classifier achieved an area under the receiver operating characteristic score of 0.69 and a Matthews correlation coefficient of 0.25.
• A Longformer-based transformer model did not significantly outperform the XGBoost baseline.
• Analysis indicated that expressions of satisfaction within chats correlated with helpfulness ratings, while rejection of exercises predicted unhelpfulness.

Conclusions:

• Chat interactions contain valuable data for NLP-based prediction of perceived service quality.
• Further research, including randomized trials, is needed to confirm if these predictive models lead to tangible service improvements.
• Simpler models like XGBoost can be effective, and comparing them against complex pretrained models is essential to avoid unnecessary complexity.