G2Basy: A framework to improve the RNN language model and ease overfitting problem.

Area of Science:

• Artificial Intelligence
• Natural Language Processing
• Machine Learning

Background:

• Recurrent neural networks (RNNs) are effective for language modeling but face significant overfitting challenges as network scales increase.
• Existing methods for improving RNN performance often struggle with the escalating complexity of modern language models.

Purpose of the Study:

• To introduce G2Basy, a framework designed to accelerate the training of recurrent neural networks and mitigate overfitting.
• To enhance the efficiency and performance of language models through innovative training techniques.

Main Methods:

• Developed G2Basy, a framework employing a gradient increasing and decreasing technique to dynamically adjust training batch size and input dropout.
• Implemented a pretrained word embedding initialization using "artificial features" and introduced adaptive optimizers with varying learning rates.
• Experimented on the Penn Treebank and WikiText-2 corpora to evaluate the framework's effectiveness.

Main Results:

• G2Basy significantly speeds up the training process and improves model performance compared to benchmark models of similar scale.
• The framework demonstrates superior results on the more complex WikiText-2 dataset compared to the Penn Treebank.
• Achieved comparable results to state-of-the-art models using significantly smaller network scales and fewer training epochs.

Conclusions:

• G2Basy offers an effective solution for accelerating RNN training and addressing overfitting in large-scale language models.
• The proposed gradient adjustment technique and embedding initialization contribute to enhanced model performance and efficiency.
• The framework shows promise for further improvements in natural language processing tasks, particularly with complex datasets.