Sparse low-rank separated representation models for learning from data

Area of Science:

• Machine Learning
• Numerical Analysis
• Data Science

Background:

• Learning multivariate functions from scattered data is crucial in many scientific fields.
• High-dimensional problems pose challenges for traditional function approximation methods.
• Existing sparse low-rank separated representation (SSR) models face convergence issues with standard training algorithms like alternating least-squares (ALS).

Purpose of the Study:

• To develop efficient and convergent training algorithms for sparse low-rank separated representation (SSR) models.
• To address the convergence difficulties of existing methods, particularly for models with rank greater than 1.
• To enhance the well-posedness of function approximation problems through sparsity constraints.

Main Methods:

• Supplementing the SSR model with sparsity constraints to ensure approximation problem well-posedness.
• Proposing two novel training algorithms: cyclic coordinate descent and block coordinate descent (BCD).
• Analyzing the convergence properties and computational complexity of the proposed algorithms, comparing them to ALS.

Main Results:

• The block coordinate descent (BCD) algorithm guarantees convergence to a Nash equilibrium point.
• The proposed algorithms exhibit linear computational cost scaling with model parameters, outperforming ALS.
• Numerical studies on synthetic and real-world regression datasets demonstrate the effectiveness of the SSR model.

Conclusions:

• The proposed sparse SSR model, trained with efficient algorithms like BCD, offers a promising approach for high-dimensional machine learning.
• The integration of sparsity constraints enhances model stability and training reliability.
• The developed methods provide a computationally efficient and convergent solution for complex function learning tasks.