Approximation by non-symmetric networks for cross-domain learning

Area of Science:

• Machine Learning
• Approximation Theory
• Neural Networks

Background:

• Machine learning research has extensively studied the approximation capabilities (expressive power) of various models, including neural networks and kernel-based methods, for over three decades.
• Existing research often relies on symmetric or positive definite kernels, limiting the scope of analysis for certain applications.

Purpose of the Study:

• To develop a general framework for studying the approximation capabilities of kernel-based networks utilizing non-symmetric kernels.
• To extend the analysis beyond traditional positive definite kernels, incorporating generalized translation networks and rotated zonal function kernels.
• To obtain approximation accuracy estimates for functions in Sobolev classes using ReLU networks, particularly when the smoothness parameter 'r' is non-integer.

Main Methods:

• Introduced a generalized approach to analyze kernel-based networks, moving beyond singular value decomposition for non-symmetric kernels.
• Considered a family of kernels, including generalized translation networks and rotated zonal function kernels.
• Derived uniform approximation accuracy estimates for functions in Sobolev classes using ReLU networks with non-integer smoothness parameters.

Main Results:

• Established a general method for analyzing approximation capabilities of kernel-based networks with non-symmetric kernels.
• Obtained specific accuracy estimates for uniform approximation of functions in Sobolev classes by ReLU networks, even for non-integer smoothness.
• Demonstrated the applicability of the general results to functions with low smoothness relative to the input space dimension.

Conclusions:

• The proposed general approach effectively analyzes kernel-based networks with non-symmetric kernels, expanding the theoretical understanding of their approximation power.
• The findings provide valuable insights into the approximation accuracy of ReLU networks for functions with varying smoothness properties.
• This work contributes to the theoretical foundations of machine learning, with potential implications for invariant learning, transfer learning, and advanced imaging techniques.