Quantifying facial affect changes in psychotic disorders with machine learning

Reduced facial expressivity is a key component of the negative symptoms of primary psychotic disorders such as schizophrenia. Facial expressivity in psychotic disorders is typically assessed through rating scales. We instead aimed to use machine learning and systems modelling to evaluate facial emotions.

Video data were acquired while 48 participants with a history of psychosis and 40 controls viewed a stand-up comedy video. Facial action unit time series were extracted from these video recordings with OpenFace software. Time series were transformed into the time-frequency domain with the continuous wavelet transform. A hidden Markov model (HMM) identified 8 dynamic facial affective patterns, each comprising a unique combination of action units oscillating at characteristic time scales. Individuals' facial expressions transitioned between these dynamic patterns as they viewed the naturalistic video stimulus.

Individuals with psychosis had reduced mean activation in positively valenced action units, reduced responsivity to joke punchlines, and increased dynamic activity in negative affect facial muscles. Clinical participants spent less time in a negatively valenced HMM state and more time in a low facial activity state. Time spent in the negatively valenced state was significantly associated with positive symptoms, while persistence of the low facial activity state was associated with negative symptoms.

These findings demonstrate that facial affect changes in psychotic disorders are multi-faceted, involving static and dynamic changes. Quantification of disorder-specific facial changes using machine learning may point to avenues for improved diagnostic differentiation and treatment monitoring of negative symptoms.