To Softmax, or not to Softmax: that is the question when applying Active Learning for Transformer Models
arxiv(2022)
摘要
Despite achieving state-of-the-art results in nearly all Natural Language
Processing applications, fine-tuning Transformer-based language models still
requires a significant amount of labeled data to work. A well known technique
to reduce the amount of human effort in acquiring a labeled dataset is
Active Learning (AL): an iterative process in which only the minimal
amount of samples is labeled. AL strategies require access to a quantified
confidence measure of the model predictions. A common choice is the softmax
activation function for the final layer. As the softmax function provides
misleading probabilities, this paper compares eight alternatives on seven
datasets. Our almost paradoxical finding is that most of the methods are too
good at identifying the true most uncertain samples (outliers), and that
labeling therefore exclusively outliers results in worse performance. As a
heuristic we propose to systematically ignore samples, which results in
improvements of various methods compared to the softmax function.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要