Robust Stochastic Gradient Descent With Student-t Distribution Based First-Order Momentum

Area of Science:

• Machine Learning
• Deep Learning
• Optimization

Background:

• Deep neural networks achieve success using stochastic gradient descent (SGD) methods.
• Machine learning algorithms struggle with noisy data, particularly in robotics, obscuring true patterns.
• Robustness against outliers is crucial for reliable performance in noisy environments.

Purpose of the Study:

• To develop a novel SGD optimization method with inherent robustness to noise.
• To improve the performance of deep learning models in the presence of data imperfections.
• To enhance outlier detection and rejection within optimization algorithms.

Main Methods:

• Proposing a new SGD optimization method based on the robust student-t distribution.
• Integrating the core idea into existing SGD algorithms, including Adam, creating t-Adam.
• Evaluating the performance of the proposed method across diverse machine learning tasks.

Main Results:

• The proposed t-Adam algorithm demonstrates superior robustness against noisy data compared to standard Adam.
• Integrated methods show improved performance over their original versions on regression, classification, and reinforcement learning tasks.
• The student-t distribution effectively enhances outlier detection and data handling in optimization.

Conclusions:

• The novel student-t distribution-based approach significantly improves the robustness of SGD optimization.
• t-Adam and related methods offer a powerful solution for deep learning in noisy conditions.
• This work advances optimization techniques for more reliable machine learning applications.