Insights Into Algorithms for Separable Nonlinear Least Squares Problems.

Area of Science:

• Numerical analysis
• Optimization algorithms
• Machine learning

Background:

• Separable nonlinear least squares (SNLLS) problems are crucial in machine learning, computer vision, and signal processing.
• Existing algorithms like joint optimization, alternated least squares (ALS), embedded point iterations (EPI), and variable projection (VP) are used to solve SNLLS problems.
• The intrinsic relationships among these SNLLS algorithms remain unclear in the literature.

Purpose of the Study:

• To elucidate the intrinsic relationships among various algorithms for solving separable nonlinear least squares (SNLLS) problems.
• To analyze the convergence and robustness of these SNLLS algorithms.
• To investigate Kaufman's conjecture through the analysis of the variable projection (VP) algorithm.

Main Methods:

• Derivation of relationships between variable projection (VP), embedded point iterations (EPI), and alternated least squares (ALS) algorithms.
• Convergence and robustness analysis of selected SNLLS algorithms.
• Numerical experiments for image restoration, time series fitting (RBF-AR model), and bundle adjustment.

Main Results:

• Established clear relationships between VP, EPI, and ALS algorithms for SNLLS problems.
• Demonstrated the convergence and robustness characteristics of the investigated algorithms.
• Provided a negative answer to Kaufman's conjecture based on VP algorithm analysis.
• Showcased comparative performance of algorithms across diverse applications.

Conclusions:

• The study provides novel insights into the connections between major SNLLS algorithms.
• Algorithm performance varies depending on the specific application, as shown in numerical experiments.
• The findings contribute to a better understanding and selection of appropriate algorithms for SNLLS problems.