Evaluating the utility of large language models for detecting and simulating language dysfunction.

Area of Science:

• Computational linguistics
• Neuroscience
• Artificial intelligence

Background:

• Language dysfunction, such as aphasia, presents challenges in data collection and analysis.
• Large language models (LLMs) offer potential for simulating and analyzing language patterns.

Purpose of the Study:

• To investigate the utility of LLMs in simulating and classifying language dysfunction, specifically aphasia.
• To evaluate LLMs' ability to approximate surface-level linguistic features of individuals with aphasia.
• To assess if synthetic data generated by LLMs can support classification tasks.

Main Methods:

• Three studies were conducted: (1) LLM-generated synthetic utterances were rated by humans for agrammatic features.
• (2) Fine-tuned LLMs were assessed on classification tasks (agrammatic feature detection, LLM-derived surprisal index classification) against a logistic regression baseline.
• (3) Exploratory analysis of LLM-based aphasia severity prediction.

Main Results:

• Human raters had difficulty distinguishing AI-generated from human-produced agrammatic speech, suggesting captured surface-level features but with low inter-rater agreement.
• A classical machine learning model outperformed LLMs in detecting agrammatic utterances, while fine-tuned LLMs showed advantages in approximating the LLM-derived surprisal index.
• LLMs, especially when fine-tuned on synthetic data, demonstrated the ability to learn task-relevant patterns of language dysfunction.

Conclusions:

• LLMs show potential as tools for data evaluation and generation in the study of language dysfunction.
• Transformer-based models can learn patterns of disordered language, offering insights into linguistic features.
• Further development of LLMs may enhance their utility in clinical and research applications for aphasia and other language disorders.