Systematic evaluation of semantic similarity across language models for schizophrenia spectrum disorders

Semantic similarity features derived from language embeddings have been proposed as markers of schizophrenia spectrum disorders (SSD), but prior studies have reported inconsistent directions of effect and limited evidence regarding their relationship to specific dimensions of thought and language disturbance. We systematically evaluated how language model architecture and preprocessing strategy influence semantic similarity signals in SSD.

Speech samples from 228 participants (152 SSD, 76 healthy volunteers) were analyzed using six embedding models: GloVe, LSA, Word2Vec, RoBERTa, GPT-2, and Llama. Semantic similarity features were computed across three preprocessing levels (verbatim, disfluency-removed, and disfluency-removed with stopword filtering) using k-interword, moving-window, and sentence-to-sentence comparisons, summarized with various statistical descriptors. Features were compared between SSD and healthy volunteers and examined for associations with dimensions of thought and language disturbance, including inefficiency, incoherence, and impaired expressivity.

SSD was predominantly associated with increased semantic similarity across models and comparison strategies. Model performance varied by preprocessing. RoBERTa performed best on minimally processed transcripts, yielding the greatest number of diagnosis-associated features, whereas GPT-2 achieved the largest diagnostic effect size after maximal preprocessing. GPT-2 showed the strongest signal for inefficiency and incoherence, while RoBERTa yielded the greatest number of associations with impaired expressivity.

SSD diagnosis and its dimensions of thought and language abnormalities are characterized primarily by greater semantic proximity in open-ended speech. Optimal preprocessing depends on model architecture and the clinical dimension of interest, highlighting the importance of model-and phenotype-specific analytic choices in computational studies of thought and language disturbance in SSD.