Explaining pretrained language models' understanding of linguistic structures using construction grammar.

Area of Science:

• Cognitive Linguistics
• Computational Linguistics
• Natural Language Processing

Background:

• Construction Grammar (CxG) views language as comprising constructions, which integrate syntax and semantics.
• Assessing the linguistic knowledge of pretrained language models (PLMs) is crucial for advancing AI.
• The English comparative correlative (CC) is a complex construction requiring both syntactic and semantic understanding.

Purpose of the Study:

• To evaluate the ability of state-of-the-art PLMs to process the English comparative correlative (CC).
• To investigate the compatibility of Construction Grammar principles with PLM capabilities.
• To determine if PLMs understand the syntactic structure versus the semantic meaning of CCs.

Main Methods:

• Experiments were conducted using BERT, RoBERTa, and DeBERTa as example PLMs.
• A syntactic probe assessed the models' accuracy in classifying CC structures.
• A semantic application task evaluated the models' comprehension of CC meaning.

Main Results:

• All tested PLMs (BERT, RoBERTa, DeBERTa, OPT) successfully recognized the syntactic structure of the CC.
• The investigated PLMs demonstrated significant limitations in understanding the semantic meaning of the CC.
• Performance on syntactic classification contrasted sharply with failures in semantic application.

Conclusions:

• PLMs exhibit proficiency in recognizing grammatical structures but lack deep semantic understanding.
• Current PLMs show substantial shortcomings in core linguistic knowledge, despite advancements in NLP tasks.
• Further research is needed to bridge the gap between syntactic processing and semantic comprehension in PLMs.