Homotopy Relaxation Training Algorithms for Infinite-Width Two-Layer ReLU Neural Networks.

Area of Science:

• Machine Learning
• Deep Neural Networks
• Computational Mathematics

Background:

• Traditional deep learning training methods can be slow and computationally intensive.
• Activation functions play a crucial role in neural network performance.
• Understanding training dynamics through tools like the Neural Tangent Kernel (NTK) is vital for optimization.

Purpose of the Study:

• To present a novel training algorithm, the Homotopy Relaxation Training Algorithm (HRTA), for accelerating deep neural network training.
• To introduce a homotopy activation function and a homotopy parameter relaxation technique to enhance training efficiency.
• To analyze the effectiveness of HRTA within the Neural Tangent Kernel (NTK) framework.

Main Methods:

• Developed the Homotopy Relaxation Training Algorithm (HRTA).
• Constructed a homotopy activation function connecting linear and $$ activation functions.
• Implemented a homotopy parameter relaxation technique for refined training.
• Analyzed HRTA's convergence properties using the Neural Tangent Kernel (NTK).

Main Results:

• HRTA significantly accelerates the training process compared to traditional methods.
• The algorithm demonstrates improved convergence rates, particularly within the NTK context.
• Experimental results validate theoretical findings, especially for wider neural networks.
• The proposed method shows promise for various activation functions and deep network architectures.

Conclusions:

• The Homotopy Relaxation Training Algorithm (HRTA) offers a significant advancement in accelerating deep learning training.
• HRTA provides enhanced convergence rates and demonstrates broad applicability.
• This novel approach has the potential to be extended to other activation functions and deep neural network architectures.