Training Neural Networks by Lifted Proximal Operator Machines

Area of Science:

• Machine Learning
• Optimization
• Deep Learning

Background:

• Gradient-based methods for neural network training can suffer from vanishing or exploding gradients.
• Existing methods may struggle with various non-decreasing activation functions.
• Efficient and stable training of deep neural networks remains a key challenge.

Purpose of the Study:

• To introduce a novel training method, the Lifted Proximal Operator Machine (LPOM), for fully-connected feed-forward neural networks.
• To develop a gradient-free training approach that circumvents common gradient-related issues.
• To enable efficient and stable training across diverse network architectures and datasets.

Main Methods:

• Representing activation functions as proximal operators and incorporating them as penalties in the objective function.
• Utilizing a block coordinate descent (BCD) method due to the block multi-convexity of the LPOM formulation.
• Developing both asynchronous and synchronous parallel BCD methods for LPOM to enhance computational speed.

Main Results:

• LPOM demonstrates convergence guarantees for its block coordinate descent solver.
• The method effectively avoids gradient vanishing/exploding issues and handles various activation functions.
• Parallel implementations (asynchronous and synchronous) show efficient scalability and performance, particularly in autoencoder training.

Conclusions:

• LPOM offers a robust, gradient-free alternative for training feed-forward neural networks.
• The block multi-convex formulation and BCD approach provide a stable and efficient training mechanism.
• Parallel LPOM variants achieve fast convergence and superior performance, highlighting its practical applicability in deep learning.