Community size rather than grammatical complexity better predicts Large Language Model accuracy in a novel Wug Test.

Area of Science:

• Computational Linguistics
• Artificial Intelligence
• Psycholinguistics

Background:

• Large Language Models (LLMs) exhibit complex linguistic behaviors, sparking debate on their true abilities.
• Assessing LLM morphological generalization to novel words is crucial for understanding their linguistic competence.
• Previous research offers mixed findings on factors influencing LLM performance.

Purpose of the Study:

• To investigate LLM performance on morphological generalization using novel words.
• To compare LLM accuracy with human speakers across diverse languages.
• To determine if linguistic complexity or community size/data availability primarily shapes LLM accuracy.

Main Methods:

• A multilingual Wug Test adaptation was used to assess six LLMs.
• Models were tested on four languages: Catalan, English, Greek, and Spanish.
• LLM performance was benchmarked against human speaker accuracy.

Main Results:

• LLMs demonstrated human-like accuracy in generalizing morphological processes to unseen words.
• Model accuracy correlated more strongly with language community size and data availability than structural complexity.
• Higher accuracy was observed in languages with larger speaker communities and greater digital resources (e.g., Spanish, English) compared to less-resourced ones (e.g., Catalan, Greek).

Conclusions:

• LLM linguistic performance appears primarily driven by the quantity and richness of training data.
• Sensitivity to grammatical complexity does not seem to be the main driver of LLM accuracy.
• LLMs may exhibit a superficial resemblance to human linguistic competence, rather than genuine understanding.