Scaling laws and representation learning in simple hierarchical languages: Transformers versus convolutional...

Area of Science:

• Computational linguistics
• Machine learning theory
• Deep learning architectures

Background:

• Neural language models (NLMs) are trained on next-token prediction tasks.
• Understanding how NLMs acquire linguistic structure is a key research question.
• Previous work established a theory of representation learning based on data correlations.

Purpose of the Study:

• To derive theoretical scaling laws for NLMs trained on synthetic data.
• To investigate the impact of model architecture on learning linguistic structure.
• To extend existing representation learning theory to account for architectural differences.

Main Methods:

• Developed theoretical scaling laws for neural network performance.
• Utilized synthetic datasets from the random hierarchy model (RHM).
• Extended representation learning theory to incorporate architectural variations.

Main Results:

• Convolutional networks show faster performance scaling than transformer models.
• Architectural biases in neural scaling laws are identified.
• Empirically validated predictions regarding network performance.

Conclusions:

• Model architecture significantly influences how NLMs learn linguistic structure.
• The alignment between network architecture and data's statistical properties is crucial.
• Findings clarify the interplay between architecture, data, and representation learning in NLMs.