Breaking the Softmax Bottleneck: A High-Rank RNN Language Model
international conference on learning representations, 2018.
We formulate language modeling as a matrix factorization problem, and show that the expressiveness of Softmax-based models (including the majority of neural language models) is limited by a Softmax bottleneck. Given that natural language is highly context-dependent, this further implies that in practice Softmax with distributed word embed...More
PPT (Upload PPT)