Analysis of argument structure constructions in the large language model BERT.

Area of Science:

• Computational Neuroscience
• Cognitive Science
• Natural Language Processing

Background:

• Understanding linguistic construction processing in the brain is key to cognitive neuroscience.
• Previous studies analyzed Argument Structure Constructions (ASCs) using Long Short-Term Memory (LSTM) networks.

Purpose of the Study:

• Investigate ASC processing and representation in the BERT language model.
• Extend previous analyses of ASCs in recurrent neural networks.

Main Methods:

• Utilized a custom GPT-4 generated dataset of 2000 sentences across four ASC types.
• Analyzed BERT token embeddings across 12 layers using Multidimensional Scaling (MDS), t-SNE, and Generalized Discrimination Value (GDV).
• Trained feedforward classifiers (probes) to predict ASC categories and analyzed attention weights using Fisher Discriminant Ratio (FDR).

Main Results:

• CLS token embeddings showed best ASC clustering in BERT layers 2-4.
• Object (OBJ) token embeddings increased substantially in clustering from layer 10, with high FDR scores.
• Probe accuracies exceeded 90% from layer 2, indicating latent construction information not evident from GDV alone.

Conclusions:

• BERT exhibits complex, layered processing of linguistic constructions, with differences from LSTMs.
• Object and verb tokens are critical for differentiating ASCs in BERT.
• Transformer and recurrent models show potential for mirroring human brain's linguistic processing.