Rich data drive generalization: Lessons from machine learning for linguistics and cognitive science

Area of Science:

• Natural Language Processing
• Machine Learning
• Artificial Intelligence

Background:

• Learning systems' generalization is sensitive to the diversity of training data.
• Variation in data, even along orthogonal axes, can influence performance.
• Prior linguistic models differed from current ones in data handling.

Purpose of the Study:

• To explore the impact of data richness on learning system generalization.
• To differentiate current language models from previous linguistic models based on data characteristics.
• To identify potential reasons for remaining linguistic data inefficiency in modern models.

Main Methods:

• Analysis of learning system generalization across diverse datasets.
• Comparative study of data variation handling in current versus prior linguistic models.
• Theoretical argumentation on the role of data richness in model performance.

Main Results:

• Data richness is a key factor affecting generalization.
• Diversity in training data, including orthogonal variations, is crucial.
• Data richness distinguishes current language models (LMs) from prior linguistic models.

Conclusions:

• The diversity of variation within training data is critical for robust generalization in learning systems.
• Data richness is a defining feature of contemporary language models.
• Despite advancements, linguistic data inefficiency in LMs may stem from limitations in data richness exploitation.