Explaining neural scaling laws

Area of Science:

• Machine Learning
• Deep Learning Theory
• Computational Neuroscience

Background:

• Trained deep neural networks exhibit predictable power-law scaling in performance loss concerning dataset size and model parameters.
• Existing research acknowledges these scaling laws but lacks a unified theoretical framework explaining their origins and interconnections.

Purpose of the Study:

• To propose a unifying theory explaining the origins of and connections between power-law scaling laws in deep neural networks.
• To identify and characterize distinct scaling regimes based on variance and resolution limitations.
• To provide a taxonomy for classifying these scaling behaviors and understand their underlying mechanisms.

Main Methods:

• Theoretical analysis connecting scaling laws to variance-limited and resolution-limited behaviors.
• Modeling deep neural networks as resolving a smooth data manifold, particularly in the large width limit.
• Empirical validation using large random feature models, pretrained models, and standard architectures across various datasets.

Main Results:

• Identification of four scaling regimes: variance-limited and resolution-limited for both dataset and model size.
• Demonstration that variance-limited scaling arises from well-behaved infinite data/width limits.
• Evidence for a duality between large width and large dataset resolution-limited scaling exponents, linked to kernel spectra.

Conclusions:

• The proposed theory successfully explains and connects observed deep learning scaling laws.
• The identified four scaling regimes provide a taxonomy for understanding performance improvements.
• Insights into the microscopic origins of scaling exponents and their relationships, highlighting different mechanisms driving loss reduction.