Analysis of argument structure constructions in a deep recurrent language model.

Area of Science:

• Cognitive Computational Neuroscience
• Computational Linguistics
• Artificial Intelligence

Background:

• Understanding neural language processing is key to cognitive neuroscience.
• Previous work analyzed Argument Structure Constructions (ASCs) in BERT.
• This study investigates ASCs in a simpler, brain-constrained recurrent neural network.

Purpose of the Study:

• To explore the representation and processing of four ASCs in a Long Short-Term Memory (LSTM) network.
• To determine if simpler neural architectures can form abstract, construction-level representations.
• To support the hypothesis that hierarchical linguistic structure emerges from prediction-based learning.

Main Methods:

• Trained an LSTM on 2,000 custom GPT-4-generated sentences.
• Analyzed internal hidden layer activations using Multidimensional Scaling (MDS) and t-Distributed Stochastic Neighbor Embedding (t-SNE).
• Quantified cluster separation using Generalized Discrimination Value (GDV).

Main Results:

• Distinct clusters for the four ASCs were observed across all LSTM hidden layers.
• The strongest separation of ASC representations occurred in the final hidden layer.
• Results align with previous findings in large language models.

Conclusions:

• Even simple recurrent neural networks can develop abstract representations of linguistic constructions.
• Findings support the emergence of hierarchical linguistic structure through prediction-based learning.
• Future work will compare model representations with neuroimaging data.