Minorization-Maximization for Ratio Sum Linear Discriminant Analysis

Linear discriminant analysis (LDA) is a widely used supervised dimension reduction method. Recently, an important variant of LDA called ratio sum LDA (RSLDA) has been proposed. RSLDA is notable for its unique ability to maximize the discriminative power of each feature in the subspace. To solve the RSLDA optimization problem, the existing equivalent transformation method converts the original problem into an equivalent one composed of two subproblems, with an auxiliary parameter introduced to ensure their convexity. In this way, the solution can be obtained via a double-loop (DL) algorithm. However, it has been found that convergence of this algorithm can be painfully slow. In this brief, we propose a novel minorization-maximization (MM) procedure that yields a simple and efficient MM algorithm for the RSLDA problem. The MM requires only a single loop, and its main novelty lies in its natural ability to determine the auxiliary parameter in a manner that achieves the fastest convergence. Furthermore, the MM is extended to solve the kernel RSLDA (KRSLDA) problem. The experimental results show that the proposed MM algorithm converges significantly faster than the DL algorithm, providing a solution with a higher objective function value in less time. Importantly, RSLDA and KRSLDA with MM each generally achieve superior or comparable classification accuracy compared to their respective DL counterparts and several related methods.