Improving Sparsity and Scalability in Regularized Nonconvex Truncated-Loss Learning Problems | JoVE Visualize

Area of Science:

Machine Learning
Optimization Algorithms
Support Vector Machines

Background:

Truncated regularized loss support vector machines (SVMs) offer advantages in robustness and scalability by reducing excessive support vectors (SVs).
Current state-of-the-art solvers like difference convex algorithm and concave-convex procedure exhibit limited sparsity promotion for truncated losses, particularly the regularized loss, and poor scalability for large datasets.
These limitations hinder the effective application of truncated loss SVMs in real-world, large-scale machine learning scenarios.

Purpose of the Study:

To address the limitations of existing solvers for truncated regularized loss SVMs.
To develop a novel multistage scheme that enhances sparsity promotion and improves scalability for large-scale problems.
To provide a clear interpretation of support vectors (SVs) and outliers within the proposed framework.

Main Methods:

A general multistage scheme is proposed to solve nonconvex truncated loss minimization problems.
The scheme decomposes the problem into a sequence of convex subproblems, with outliers identified and removed in advance.
Specific algorithmic improvements include a linear multistep approach using coordinate descent and a kernel-based method leveraging Karush-Kuhn-Tucker conditions for efficient outlier identification.

Main Results:

The proposed general multistage algorithm demonstrates significant sparsity promotion, especially for the truncated regularized loss.
The linear multistep and kernel algorithms show substantial improvements in scalability for large-scale datasets.
Experimental comparisons confirm the superiority of the proposed methods in terms of both sparsity and efficiency.

Conclusions:

The developed multistage scheme effectively overcomes the sparsity and scalability limitations of previous solvers for truncated regularized loss SVMs.
The proposed algorithms offer a robust and efficient solution for large-scale machine learning tasks utilizing truncated loss functions.
This work provides a valuable advancement in the field of optimization for support vector machines, enhancing their practical applicability.