Foreseeing the Benefits of Incidental Supervision.

EMNLP(2021)

引用 11|浏览77
暂无评分
摘要
Learning theory mostly addresses the standard learning paradigm, assuming the availability of complete and correct supervision signals for large amounts of data. However, in practice, machine learning researchers and practitioners acquire and make use of a range of {\em incidental supervision} signals that only have statistical associations with the gold supervision. This paper addresses the question: {\em Can one quantify models' performance when learning with such supervision signals, without going through an exhaustive experimentation process with various supervision signals and learning protocols?} To quantify the benefits of various incidental supervision signals, we propose a unified PAC-Bayesian Informativeness measure (PABI), characterizing the reduction in uncertainty that incidental supervision signals provide. We then demonstrate PABI's use in quantifying various types of incidental signals such as partial labels, noisy labels, constraints, cross-domain signals, and some combinations of these. Experiments on named entity recognition and question answering show that PABI correlates well with learning performance, providing a promising way to determine, ahead of learning, which supervision signals would be beneficial.
更多
查看译文
关键词
supervision,benefits
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要