A distributed optimisation framework combining natural gradient with Hessian-free for discriminative sequence...

Area of Science:

• Machine Learning
• Artificial Intelligence
• Speech Processing

Background:

• Neural network training often requires substantial computational resources and numerous parameter updates.
• Existing optimization methods may not be sufficiently efficient for complex acoustic models in speech recognition.

Purpose of the Study:

• To present a novel Natural Gradient and Hessian-Free (NGHF) optimization framework for efficient, distributed neural network training.
• To enhance parameter update efficiency and model performance in acoustic modeling for automatic speech recognition.

Main Methods:

• The Natural Gradient and Hessian-Free (NGHF) framework combines natural gradient (NG) with Hessian-free (HF) methods using the linear conjugate gradient (CG) algorithm.
• A novel solution addresses numerical issues in CG, enabling effective parameter updates with significantly fewer iterations (5-8 vs. 200).
• A new preconditioning approach is introduced to boost CG iteration progress for models with shared parameters.

Main Results:

• NGHF was applied to lattice-based discriminative sequence training for hybrid Hidden Markov Model acoustic models using Recurrent Neural Network, Long Short-Term Memory, and Time Delay Neural Network architectures.
• Experiments on the Multi-Genre Broadcast dataset demonstrated that NGHF achieved greater word error rate reductions compared to standard Stochastic Gradient Descent and Adam.
• The NGHF framework required orders of magnitude fewer parameter updates than conventional methods.

Conclusions:

• The NGHF framework offers a more efficient and effective optimization strategy for training neural networks, particularly in the context of acoustic modeling for automatic speech recognition.
• NGHF presents a significant advancement over existing optimization techniques, demonstrating superior performance with reduced computational demands.