Large language model linguistic perplexity in childhood onset psychosis: unique features and developmental trends

Child and early adolescent onset psychosis (COP) is associated with subtle changes in language linked to thought disorder, a key contributor to functional impairment. Large language models (LLM) can detect deviations from expected language patterns by jointly analyzing sentence structure and word choice. This integrated information is captured by measures such as (pseudo)-perplexity, which quantify how difficult it is for an LLM to predict individual words given the surrounding linguistic context. The objective was to determine if perplexity measures change with age and whether they are altered in COP.

This study tested for a difference in perplexity in COP cases (N = 23) mean age 12.78 years as compared to controls (N = 15) mean age 11.67 years. Extensive manually transcribed interviews were analyzed (controls: 3,829; cases: 6,579 mean words).

Perplexity derived from Large Language Model Meta AI (LLaMA) had a significant negative correlation with age in cases but not in controls. Pseudo-perplexity derived from Bidirectional Encoder Representations from Transformers (BERT) did not have a significant correlation with age in either group. Group differences were evaluated using generalized linear models with (pseudo)-perplexity as the dependent variable, case status as the predictor and age and number of words as covariates. The model predicting perplexity was significant and case status significantly predicted perplexity. In contrast, the model predicting pseudo-perplexity was not significant.

These differences in perplexity are interpreted as reflecting an altered developmental trajectory in the real-time semantic and syntactic planning that directs the flow of language in individuals with COP.