Precision Machine Learning.

Area of Science:

• Scientific Machine Learning
• Computational Science
• Data Science

Background:

• Machine learning (ML) models require high precision for scientific applications.
• Classical function approximation methods face challenges with scaling and high-dimensional data.

Purpose of the Study:

• To explore unique considerations for fitting ML models to high-precision scientific data.
• To compare the scalability of various function approximation methods with increasing parameters and data.
• To investigate the optimization challenges of neural networks (NNs) in the high-precision regime.

Main Methods:

• Empirical comparison of function approximation methods, including neural networks (NNs).
• Analysis of model scaling with increasing parameters and data.
• Study of NN loss landscapes and optimization challenges in the high-precision regime.
• Development of novel training techniques for NNs.

Main Results:

• NNs outperform classical methods on high-dimensional data, likely by exploiting modular structures.
• NNs trained with common optimizers are less effective in low-dimensional cases.
• Novel training tricks enable NNs to achieve extremely low loss, near numerical precision limits.

Conclusions:

• NNs offer significant advantages for high-precision scientific ML, particularly in high dimensions.
• Addressing optimization challenges in low-dimensional settings is crucial for broadening NN applicability.
• Developed training techniques enhance NN performance in the high-precision regime, pushing computational boundaries.