LightGAN : An Adversarial Approach to Natural Language Generation at a Large Scale

NATURAL language generation is important in the today’s environment of digital assistants. It is, however, a difficult task to generate language that makes sense. Traditional approaches like n-grams become successful when the value for n is large (generally about 5) and in that case, the text generated tends to repeat itself or simply output a sentence that it was trained on. This defeats the purpose of language generation as a method of creating new, novel sentences. Recently, RNNs (especially those with LSTMs) have become successful at modeling language that generally makes sense. These RNNs take large amount of memory that are dependent on several factors including the size of the vocabulary being trained on. In order to achieve state-of-the-art performance on large datasets, these vocabularies must be large. This poses a unique challenge when needing to train on resource limited systems (eg 2 gpus). While there is some work that has reduced memory requirements like sampled and hierarchal softmaxes, however we propose that looking at different LSTM designs and training paradigms will lead to state-of-the-art models in natural language generation. We will combine the work from two papers that were both sought to improve upon natural language generation. The first uses a GAN approach to training and the second uses a novel method (a new LSTM cell) to reduce the number of parameters that the network needs at the expense of extra compute time. We believe that this combined model will achieve a high level of performance as both methods have been shown to achieve state-of-the-art or nearly state-of-the-art performance on NLP benchmarks like the PTB and the Billion Word Benchmark. II. THE DATA