Orthogonalizing EM: A design-based least squares algorithm.

Area of Science:

• Statistics
• Machine Learning
• Numerical Analysis

Background:

• Least squares problems are fundamental in data analysis and statistical modeling.
• Existing methods can be computationally intensive, especially for large-scale datasets.
• Efficient algorithms are crucial for handling complex regression and penalized regression scenarios.

Purpose of the Study:

• Introduce an efficient iterative algorithm, orthogonalizing EM (OEM), for solving ordinary and penalized least squares problems.
• Demonstrate the theoretical properties and convergence of the OEM algorithm.
• Showcase the efficiency of OEM for large-scale problems.

Main Methods:

• The orthogonalizing EM (OEM) algorithm is based on a design of experiments perspective.
• It involves orthogonalizing a design matrix by adding rows and embedding it in a missing data framework.
• Theoretical properties, convergence, and convergence rates are rigorously examined.

Main Results:

• OEM converges to the Moore-Penrose generalized inverse-based least squares estimator for singular regression matrices.
• It converges to a point with grouping coherence for fully aliased regression matrices in both ordinary and penalized least squares.
• OEM demonstrates high efficiency and is considerably faster than competing methods when n >> p.

Conclusions:

• The orthogonalizing EM (OEM) algorithm provides an efficient and theoretically sound approach for least squares and penalized least squares problems.
• OEM is particularly advantageous for large-scale datasets where computational speed is critical.
• This method offers a valuable tool for statisticians and data scientists dealing with extensive regression analyses.