A Mutual Information Maximization Perspective of Language Representation Learning

d'Autume Cyprien de Masson
d'Autume Cyprien de Masson
Yu Lei
Yu Lei
Cited by: 29|Views96
Weibo:
We provided a unifying view of classical and modern word embedding models and showed how they relate to popular representation learning methods used in other domains

Abstract:

We show state-of-the-art word representation learning methods maximize an objective function that is a lower bound on the mutual information between different parts of a word sequence (i.e., a sentence). Our formulation provides an alternative perspective that unifies classical word embedding models (e.g., Skip-gram) and modern contextu...More

Code:

Data:

0
Full Text
Bibtex
Weibo
Your rating :
0

 

Tags
Comments