(R)NNs too expressive?: No relationship between neural architectures' theoretical expressivity and their empirical...

Area of Science:

• Computational Linguistics
• Cognitive Science
• Artificial Intelligence

Background:

• Recurrent neural networks (RNNs) often learn unattested phonological patterns.
• This over-generation is hypothesized to stem from their high expressive capacity exceeding human linguistic constraints.

Purpose of the Study:

• To investigate the reasons behind RNNs' tendency to over-generate phonological patterns.
• To test the hypothesis that excess model expressivity causes this over-generation.
• To compare RNNs with Convolutional Neural Networks (CNNs) on phonological pattern recognition tasks.

Main Methods:

• Comparison of RNNs and CNNs on string recognition tasks.
• Evaluation of model expressivity as a predictor of performance.
• Analysis of architectural biases, specifically position-invariance.

Main Results:

• Model expressivity did not correlate with recognition performance across different string classes.
• CNNs did not necessarily outperform RNNs due to inherently limited expressive capacity.
• CNNs' success was better explained by their inherent position-invariant biases.

Conclusions:

• The expressivity of neural network architectures does not solely determine their ability to learn attested phonological patterns.
• Position-invariant biases in models like CNNs appear crucial for recognizing certain linguistic structures.
• Further research should focus on architectural biases rather than just model complexity to understand human-like linguistic learning.