A multivariate adaptive gradient algorithm with reduced tuning efforts.

Area of Science:

• Machine Learning
• Optimization Algorithms

Background:

• Training large neural networks is computationally expensive due to a high number of parameters.
• Existing methods often require significant tuning and high per-iteration costs.

Purpose of the Study:

• To propose a multivariate adaptive gradient descent method with low per-iteration costs and fast convergence.
• To reduce the tuning effort required for training large neural networks.

Main Methods:

• Developed a multivariate adaptive gradient descent method updating parameters separately with an adaptive vector-form learning rate.
• The adaptive learning rate is based on the difference in parameters and subgradients.
• Analyzed convergence rates for various cost functions (smooth, strongly convex, non-convex, non-smooth).

Main Results:

• Demonstrated linear convergence rate for smooth and strongly convex functions in a deterministic setting.
• Showed expected gradient convergence of O(1/k) for non-convex functions in deterministic and stochastic settings.
• Proved cost function convergence of O(log(T)/T) for non-smooth strongly convex functions.
• Empirically validated effectiveness on various functions and image classification datasets, showing superior performance with minimal tuning.

Conclusions:

• The proposed adaptive gradient descent method is computationally efficient and requires minimal tuning.
• It achieves state-of-the-art performance compared to existing optimizers on benchmark tasks.
• Offers a promising alternative for training large-scale machine learning models.