Representations and generalization in artificial and brain neural networks.

Area of Science:

• Neuroscience
• Machine Learning
• Cognitive Science

Background:

• Humans and animals generalize effectively from limited data, a feat not yet matched by artificial intelligence.
• Generalization in biological and artificial deep neural networks (DNNs) is crucial for real-world applications, encompassing both in-distribution and out-of-distribution scenarios.

Purpose of the Study:

• To investigate generalization in biological and artificial deep neural networks (DNNs).
• To propose hypotheses linking neural manifold geometry and DNN learning theory to generalization capabilities.
• To bridge neuroscience, machine learning, and cognitive science through a unified methodology.

Main Methods:

• Overviewing recent progress in studying the geometry of neural manifolds, particularly in visual object recognition.
• Discussing theories connecting manifold dimension and radius to generalization capacity.
• Exploring the theory of learning in wide DNNs, including the role of weight norm regularization, network architecture, and hyperparameters.

Main Results:

• Neural manifold geometry, specifically its geometric properties, acts as an order parameter linking neural substrates to generalization.
• Theories of learning in wide DNNs provide mechanistic insights into generating desired neural representational geometries and generalization.
• Weight norm regularization, network architecture, and hyperparameters play significant roles in DNN generalization.

Conclusions:

• Geometric properties of neural manifolds are key to understanding generalization in both biological and artificial systems.
• The theory of learning in wide DNNs offers a mechanistic framework for improving generalization capabilities.
• Further research into representational drift and learning dynamics is essential for advancing AI generalization.