Fallback Variable History NNLMs: Efficient NNLMs by precomputation and stochastic training.

Area of Science:

• Computational Linguistics
• Artificial Intelligence
• Machine Learning

Background:

• Neural Networks are powerful language models but can be computationally expensive.
• Optimizing the softmax normalization layer is crucial for improving efficiency.

Purpose of the Study:

• To present a novel method for reducing computational costs in Neural Network Language Models.
• To enhance recognition speed and system performance in specific scenarios.

Main Methods:

• Developing a Neural Network that processes variable-length input contexts.
• Implementing a fallback mechanism and precomputing softmax normalization constants.
• Validating the approach on a machine translation task.

Main Results:

• The proposed method effectively emulates lower-order N-grams using a single Neural Network.
• It significantly reduces the normalization cost of the output softmax layer.
• System speed is improved without a substantial impact on performance.

Conclusions:

• The new method offers a practical solution for optimizing Neural Network Language Models.
• It balances computational efficiency with performance accuracy for specific applications.