An approximation theory perspective on machine learning.

Area of Science:

• Machine Learning
• Approximation Theory
• Mathematical Foundations of Computer Science

Background:

• Machine learning aims to build models that approximate unknown probability distributions from sampled data.
• Neural networks and kernel methods are widely used for their computational efficiency, with their approximation capabilities studied for decades.
• A disconnect exists between approximation theory and machine learning practice, impacting model generalization.

Purpose of the Study:

• To review key ideas and emerging trends in function approximation within machine learning.
• To analyze the gap between approximation theory and current machine learning practices.
• To explore recent advancements in manifold approximation and classification as signal separation.

Main Methods:

• Literature review of approximation capabilities in machine learning.
• Discussion of shallow/deep networks, manifold approximation, neural operators, and transformers.
• Examination of recent work on function approximation on unknown manifolds and classification as signal separation.

Main Results:

• Identified shortcomings in the current machine learning framework regarding generalization.
• Reviewed methods for function approximation on manifolds without explicit feature learning.
• Summarized a novel approach viewing classification as a signal separation problem.

Conclusions:

• Approximation theory is crucial for robust machine learning but is underutilized.
• Emerging trends like neural operators and manifold learning offer promising avenues.
• Further research is needed to bridge the theory-practice gap and address open problems in machine learning.