Natural language processing models for predicting treatment outcomes in internet-delivered cognitive behavioral...

Predicting treatment outcome has the potential to enhance Internet-delivered Cognitive Behavioral Therapy (ICBT). One aspect of guided ICBT is patient-therapist interaction through written messages. With Natural language processing (NLP) these could be leveraged to predict outcome; however current evidence is limited. This study investigates the predictive accuracy of NLP models for treatment outcomes and evaluates whether NLP provides additional predictive value beyond symptom variables.

Patient-therapist messages from 6613 patients undergoing 12 weeks of treatment were used to train three types of NLP models: Term Frequency-Inverse Document Frequency (TF-IDF), Bidirectional Encoder Representations from transformers (BERT), and BERT for Longer Text (BELT). These were trained both with and without symptom variables from the initial treatment period to predict post-treatment symptoms. A dummy model was also used, and a linear regression model acted as a symptoms only benchmark. Multiple imputation addressed missing data, and nested cross-validation was used.

The symptom only model performed best. Only BERT outperformed the dummy model, achieving a Root Mean Squared Error (RMSE) of 0.17 compared to RMSE of 0.18. Adding symptom variables to the BERT model significantly increased its accuracy, but not the RMSE metric. The best linear regression benchmark based on symptoms only had a BACC of 70 % (F₁-score of 0.66) which outperformed the BERT model with 60 % (F₁: 0.55) and the combined BERT plus symptoms model achieved 68 % (F₁: 0.62).

These initial findings indicate a small predictive value from patient-therapist written message interaction but added no value beyond using only symptoms to predict post-treatment symptoms. Further research is needed to refine NLP-methods for use in psychological treatment, and more accurately assess the predictive potential of text-based interactions during ICBT.