Evaluating the Performance of Transformer-based Language Models for Neuroatypical Language

Area of Science:

• Computational linguistics
• Neurodevelopmental disorders
• Artificial intelligence in healthcare

Background:

• Social communication difficulties are a core feature of autism spectrum disorder (ASD).
• These challenges can impact employment opportunities for adults with ASD.
• Targeted interventions are needed to improve pragmatic and social language skills.

Purpose of the Study:

• To develop and evaluate a transformer-based framework for identifying linguistic features in social communication.
• To analyze conversational data from adults with and without ASD during collaborative tasks.
• To assess the performance of AI models in understanding neuroatypical language patterns.

Main Methods:

• Utilized a transformer-based framework for linguistic feature identification.
• Analyzed a corpus of conversations involving adults with ASD and neurotypical individuals.
• Compared model performance on language from participants with and without ASD.

Main Results:

• The framework achieved strong overall accuracy in identifying social linguistic features.
• Model performance was significantly lower for the language of participants with ASD.
• This suggests a greater diversity of social linguistic strategies employed by individuals with ASD.

Conclusions:

• The developed framework shows promise for creating automated tools to support language interventions for ASD.
• Current large language models may struggle to fully capture the nuances of neuroatypical communication.
• Further research is needed to improve AI's ability to model diverse linguistic strategies in ASD.