Subword language modeling with neural networks

We explore the performance of several types of language mode ls on the word-level and the character-level language modelin g tasks. This includes two recently proposed recurrent neural netwo rk architectures, a feedforward neural network model, a maximum ent ropy model and the usual smoothed n-gram models. We then propose a simple technique for learning sub-word level units from th e data, and show that it combines advantages of both character and wo rdlevel models. Finally, we show that neural network based lan gu ge models can be order of magnitude smaller than compressed n-g ram models, at the same level of performance when applied to a Bro dcast news RT04 speech recognition task. By using sub-word un its, the size can be reduced even more.