Sufficient is better than optimal for training neural networks.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computational Neuroscience

Background:

• Optimization-based training is a common paradigm for neural networks.
• This approach often leads to overfitting, where models learn spurious correlations.
• Ad hoc modifications are frequently needed, suggesting fundamental limitations.

Purpose of the Study:

• To introduce a novel physics-based training method called simmering.
• To demonstrate that simmering can outperform traditional optimization-based training.
• To challenge the prevailing optimization paradigm in neural network training.

Main Methods:

• Simmering trains neural networks by systematically sampling non-optimal weights and biases.
• This creates an ensemble of models that represent the underlying phenomenon sufficiently.
• Information-geometric arguments are used to support the theoretical basis of simmering.

Main Results:

• Simmering paradoxically outperforms leading optimization-based approaches.
• The method effectively corrects overfit neural networks.
• Simmering produces more generalizable predictions compared to other overfitting mitigation techniques.

Conclusions:

• Optimization may not be the ideal paradigm for training various neural network architectures.
• Simmering offers a viable alternative, improving model generalization and robustness.
• Further research into non-optimization-based training algorithms is warranted.