Partial local entropy and anisotropy in deep weight spaces.

Area of Science:

• Machine Learning
• Deep Learning
• Optimization

Background:

• Local entropic loss functions are used in deep learning.
• Existing methods may not fully exploit the complex geometry of neural network weight spaces.

Purpose of the Study:

• To refine local entropic loss functions by applying regularization to a subset of weights.
• To introduce and analyze partial local entropies for improved neural network training.

Main Methods:

• Theoretical analysis of new loss functions.
• Experimental validation on image classification tasks using various neural network architectures (multilayer, fully connected, convolutional).

Main Results:

• Partial local entropies outperform isotropic counterparts by adapting to weight-space anisotropy.
• The study provides insights into the shape of minima found by stochastic gradient descent algorithms.
• An asymptotic dynamical regime with common cooling behavior across layers was observed at late training times.

Conclusions:

• Partial local entropies offer a more effective way to exploit the anisotropic nature of deep learning landscapes.
• The findings contribute to a better understanding of optimization dynamics in deep neural networks.