A Mutual Information Maximization Perspective of Language Representation Learning

Cyprien de Masson d'Autume
Cyprien de Masson d'Autume

ICLR, 2020.

Cited by: 3|Views521
EI
Weibo:
We provided a unifying view of classical and modern word embedding models and showed how they relate to popular representation learning methods used in other domains

Abstract:

We show state-of-the-art word representation learning methods maximize an objective function that is a lower bound on the mutual information between different parts of a word sequence (i.e., a sentence). Our formulation provides an alternative perspective that unifies classical word embedding models (e.g., Skip-gram) and modern contextual...More

Code:

Data:

Your rating :
0

 

Tags
Comments