SemGloVe: Semantic Co-Occurrences for GloVe From BERT
IEEE-ACM TRANSACTIONS ON AUDIO SPEECH AND LANGUAGE PROCESSING(2022)
摘要
GloVe learns word embeddings by leveraging statistical information from word co-occurrence matrices. However, word pairs in the matrices are extracted from a predefined local context window, which might lead to limited word pairs and potentially semantic irrelevant word pairs. In this paper, we propose SemGloVe, which distills semantic co-occurrences from BERT into static GloVe word embeddings. Particularly, we propose two models to extract co-occurrence statistics based on either the masked language model or the multi-head attention weights of BERT. Our methods can extract word pairs limited by the local window assumption, and can define the co-occurrence weights by directly considering the semantic distance between word pairs. Experiments on several word similarity datasets and external tasks show that SemGloVe can outperform GloVe.
更多查看译文
关键词
Bit error rate, Semantics, Task analysis, Speech processing, Context modeling, Mathematical models, Vocabulary, Word representations, contextual word embeddings, self-attention network, pre-trained language models
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络