Fundamental limits to learning closed-form mathematical models from data

Area of Science:

• Computational Mathematics
• Statistical Learning Theory
• Machine Learning

Background:

• Determining the learnability of a generating model from finite, noisy data is a fundamental challenge.
• Understanding the impact of noise on model inference and generalization is crucial for data-driven scientific discovery.

Purpose of the Study:

• To investigate the conditions under which a true closed-form mathematical model can be learned from a finite, noisy dataset.
• To identify the critical noise level that transitions the problem from a learnable to an unlearnable phase.
• To compare the generalization performance of probabilistic model selection against standard machine learning approaches.

Main Methods:

• Analysis of a finite and noisy dataset generated by a closed-form mathematical model.
• Investigation of model-learning feasibility across varying levels of observation noise.
• Comparative evaluation of generalization performance using probabilistic model selection and standard machine learning methods (e.g., artificial neural networks).

Main Results:

• A clear transition was observed from a low-noise phase (model learnable) to a high-noise phase (model unlearnable).
• Probabilistic model selection demonstrated optimal generalization in both low- and high-noise phases.
• Standard machine learning, including artificial neural networks, showed limitations in interpolation within the low-noise phase.

Conclusions:

• The learnability of a true mathematical model from data is critically dependent on the level of observation noise.
• Probabilistic model selection offers superior generalization capabilities compared to standard machine learning for this specific problem.
• The transition region presents significant challenges for generalization across all evaluated methods.