Can large language models be viewed as a cognitive model of human language? Not yet, regardless of reasoning...

Can large language models (LLMs) be viewed as a cognitive model of human language? Do they possess human-like language competence? To address these questions, this study takes a multifaceted approach, comparing the performance of 10 recent LLMs (n = 4000 responses) and 94 humans (n = 3760 responses) on grammaticality judgments and sentence interpretations, focusing on five linguistic phenomena that involve missing material. The analyses show that while the LLMs appeared to differentiate between grammatical/possible and ungrammatical/impossible sentences/interpretations overall, they struggled with infrequent phenomena (e.g., Gapping, Sluicing), often rejecting grammatical sentences and accepting impossible interpretations. Notably, increased size seemed to improve their performance on grammaticality judgments, but neither size nor reasoning capability improved their performance on interpretation. In contrast, humans demonstrated a clear sensitivity to these distinctions. The findings seem to align with the view that LLMs, in their current form, lack language competence and do not provide a convincing explanation of human language.