Adaptive importance sampling to accelerate training of a neural probabilistic language model

Area of Science:

• Natural Language Processing
• Machine Learning
• Computational Linguistics

Background:

• Neural network language models (NNLMs) offer improved performance over traditional n-gram models.
• NNLMs approximate word sequence probabilities, reducing errors in language modeling.
• Maximum-likelihood training for NNLMs is computationally intensive, scaling with vocabulary size.

Purpose of the Study:

• To introduce a novel method for accelerating the training of neural network language models.
• To address the computational bottleneck associated with maximum-likelihood training in NNLMs.
• To improve the efficiency of statistical language modeling.

Main Methods:

• Adaptive importance sampling (AIS) is proposed to accelerate NNLM training.
• An adaptive n-gram model is employed to track the conditional distributions generated by the neural network.
• The AIS technique optimizes the sampling process during model training.

Main Results:

• Significant speedups in training time were achieved using adaptive importance sampling.
• The proposed method effectively accelerates computations without sacrificing model accuracy.
• Experimental results demonstrate the practical benefits of AIS on standard language modeling tasks.

Conclusions:

• Adaptive importance sampling provides a computationally efficient alternative for training neural network language models.
• The integration of adaptive n-gram models with AIS enhances training speed.
• This approach offers a viable solution to the scalability challenges in NNLM training.