Piece-wise quadratic approximations of arbitrary error functions for fast and robust machine learning

Area of Science:

• Machine Learning
• Optimization Theory
• Data Science

Background:

• Traditional machine learning relies on quadratic optimization, which struggles with high-dimensional, noisy data.
• Non-quadratic error functionals (e.g., L1 norm) offer improvements but increase computational cost.
• Existing methods lack a flexible, efficient framework for arbitrary error functionals.

Purpose of the Study:

• To develop a universal and computationally efficient framework for minimizing arbitrary sub-quadratic error potentials in machine learning.
• To introduce Piece-wise Quadratic error potentials of sub-quadratic growth (PQSQ potentials) as a novel approach.
• To enhance the trade-off between computational cost and accuracy in various machine learning algorithms.

Main Methods:

• Development of a theory based on the cone of minorant functions and min-plus algebra.
• Introduction of Piece-wise Quadratic error potentials of sub-quadratic growth (PQSQ potentials).
• Design of a universal algorithm for minimizing PQSQ potentials with guaranteed convergence.

Main Results:

• PQSQ potentials provide a flexible framework applicable to diverse machine learning algorithms.
• The proposed algorithm demonstrates fast convergence to error minima.
• Demonstrated significant improvements in computational performance (orders of magnitude faster) compared to state-of-the-art methods on synthetic and real-life datasets.
• Achieved similar or better approximation accuracy.

Conclusions:

• PQSQ potentials offer a computationally efficient and accurate alternative to traditional quadratic and existing non-quadratic error minimization methods.
• The developed framework enhances the performance of data approximation and regularized/sparse regression.
• This approach represents a significant advancement in machine learning optimization, improving the cost/accuracy balance.