GenEFT: Understanding statics and dynamics of model generalization via physics-inspired effective theory

Area of Science:

• Machine Learning
• Theoretical Physics
• Graph Learning

Background:

• Neural network generalization is crucial for model performance.
• Understanding the transition between generalization and overfitting remains a challenge.
• Existing theoretical frameworks often struggle to connect with practical applications.

Purpose of the Study:

• To present GenEFT, an effective theory framework for analyzing neural network generalization.
• To investigate generalization phase transitions with increasing data size.
• To model representation learning dynamics using interacting particles.

Main Methods:

• Developed the GenEFT (Generalization Effective Field Theory) framework.
• Applied GenEFT to graph learning problems.
• Modeled latent-space representations as interacting particles ('repons').
• Analyzed phase transitions by scanning encoder and decoder learning rates.

Main Results:

• GenEFT effectively explains the statics and dynamics of neural network generalization.
• Experimental results align with information-theory-based approximations for generalization phase transitions.
• The effective theory for representation learning dynamics successfully explains the observed generalization-overfitting phase transition.

Conclusions:

• GenEFT provides a powerful physics-inspired approach to understanding machine learning.
• The framework bridges the gap between theoretical predictions and practical machine learning scenarios.
• Effective theories offer a promising direction for advancing neural network generalization research.