Natural Language Processing and Psychosis: On the Need for Comprehensive Psychometric Evaluation

Area of Science:

• Psychiatry
• Computational Linguistics
• Psychometrics

Background:

• Natural Language Processing (NLP) has shown potential in psychosis research for years.
• Clinical implementation of NLP tools is hindered by insufficient psychometric evaluation.
• While criterion and content validity are often established, test-retest reliability, divergent validity, and demographic biases require further investigation.

Purpose of the Study:

• To develop and evaluate an NLP measure for tracking clinically rated paranoia in patients with schizophrenia or bipolar disorder using smartphone video recordings.
• To address concerns regarding the psychometric properties of NLP measures in mental health research.

Main Methods:

• Recruited patients with schizophrenia or bipolar disorder for a week-long tracking study.
• Developed an NLP model using a feature set from 499 language samples to predict clinically rated paranoia.
• Employed regularized regression for modeling paranoia from video 'selfies' captured on smartphones.

Main Results:

• The NLP measure demonstrated high test-retest reliability.
• Criterion and convergent/divergent validity were contingent on moderating variables like patient location (home vs. away) and social context (alone vs. with strangers).
• Systematic racial and sex biases were identified, partly linked to recording context.

Conclusions:

• Advancing NLP for psychosis research necessitates rigorous evaluation of test-retest reliability, divergent validity, and systematic biases.
• Moderating variables significantly influence the performance of NLP measures.
• Future research should systematically address identified psychometric weaknesses and biases for robust clinical application.