Evaluating the utility of large language models for detecting and simulating language dysfunction

This study investigates the utility of large language models (LLMs) in simulating and classifying language dysfunction, with a focus on aphasia. We conducted three studies to evaluate the extent to which LLMs can approximate surface-level linguistic features of individuals with aphasia and whether synthetic data can support classification tasks when real data is limited. In Study 1, we prompted an LLM to generate synthetic utterances and used human rating surveys to evaluate whether the resulting outputs capture surface-level features associated with agrammatic language production. Results showed that approximately half of raters were unable to reliably distinguish AI-generated from human-produced agrammatic speech. This pattern suggests that the synthetic data captured some surface-level features associated with agrammatic utterances; however, low inter-rater agreement indicates substantial rater uncertainty and task ambiguity-likely reflecting the difficulty of making reliable judgments from brief, text-only excerpts-limiting conclusions about clinical fidelity. In Study 2, we assessed the performance of (fine-tuned) LLMs on two binary classification tasks: (1) detecting agrammatic features and (2) classifying utterances based on an LLM-derived surprisal index (negative log-likelihood of a token given its context). We examined models' successes and failures by comparing performance across training conditions and against a classical machine learning baseline (logistic regression) model. We found preliminary and suggestive evidence that the classical machine learning model remained superior for binary agrammatic utterances detection, however, fine-tuned LLMs showed advantages in approximating the LLM-derived surprisal index compared to their pre-trained counterparts and the baseline logistic regression model. We additionally conducted an exploratory analysis in Study 3, where we prompted an LLM for end-to-end aphasia severity prediction. Our overall results suggest that, as a proof-of-concept, transformer-based models, particularly when fine-tuned on curated synthetic data, can learn task-relevant patterns of language dysfunction. Our findings demonstrate the potential utility of LLMs in modeling not only language function (language produced by the general population) but also patterns of language dysfunction (such as those observed in aphasia), offering insights into the linguistic features of disordered language. We cautiously conclude that, with further development, LLMs may serve as useful tools for data evaluation and generation in the study of language dysfunction.