An Empirical Study of Pre-trained Embedding on Ultra-Fine Entity Typing.

The embedding generated by pre-trained models has attracted the attention of many scholars in the past few years. Most of the context-sensitive embeddings have confirmed the positive impact on some basic tasks of classification, which have only a few types. In this paper, we make an empirical comparison of different pre-trained embeddings on the task of ultra-fine entity typing which has more than 10k types. We apply 7 kinds of pre-trained embedding to the typing model to prove whether the pre-trained embedding has a positive effect. The results indicate that almost all context-sensitive pre-trained embeddings improve the performance of models using Glove. The pre-trained embedding generated by BERT achieves the best performance in the Ultra-Fine dataset and OntoNotes dataset, which shows BERT has better capability to extract finer-grained information than other pre-trained models.