InitialGAN: A Language GAN With Completely Random Initialization.

Area of Science:

• Artificial Intelligence
• Natural Language Processing
• Machine Learning

Background:

• Text generative models trained with maximum likelihood estimation (MLE) face significant exposure bias.
• Generative adversarial networks (GANs) show promise for mitigating exposure bias but often require pretraining.
• Representation modeling methods (RMMs) are less explored due to historical performance issues.

Purpose of the Study:

• To develop a language GAN that overcomes exposure bias without relying on pretraining techniques.
• To address the limitations of invalid sampling and unhealthy gradients in existing language GANs.
• To introduce a new evaluation metric for assessing the quality of generated text.

Main Methods:

• Introduced dropout sampling and a fully normalized long short-term memory (LSTM) network.
• Proposed InitialGAN, a GAN with randomly initialized parameters, leveraging these techniques.
• Developed a new evaluation metric, the least coverage rate (LCR), for generated samples.

Main Results:

• InitialGAN demonstrated superior performance compared to MLE and other existing models.
• The proposed techniques effectively addressed issues with sampling and gradients in RMMs.
• InitialGAN achieved state-of-the-art results without any pretraining.

Conclusions:

• InitialGAN successfully outperforms MLE and other language GANs, particularly in scenarios without pretraining.
• The study highlights the potential of RMMs when equipped with appropriate sampling and gradient handling.
• This work presents the first language GAN capable of outperforming MLE without pretraining.