On Sensitivity of Learning with Limited Labelled Data to the Effects of Randomness: Impact of Interactions and Systematic Choices
CoRR(2024)
摘要
While learning with limited labelled data can improve performance when the
labels are lacking, it is also sensitive to the effects of uncontrolled
randomness introduced by so-called randomness factors (e.g., varying order of
data). We propose a method to systematically investigate the effects of
randomness factors while taking the interactions between them into
consideration. To measure the true effects of an individual randomness factor,
our method mitigates the effects of other factors and observes how the
performance varies across multiple runs. Applying our method to multiple
randomness factors across in-context learning and fine-tuning approaches on 7
representative text classification tasks and meta-learning on 3 tasks, we show
that: 1) disregarding interactions between randomness factors in existing works
caused inconsistent findings due to incorrect attribution of the effects of
randomness factors, such as disproving the consistent sensitivity of in-context
learning to sample order even with random sample selection; and 2) besides
mutual interactions, the effects of randomness factors, especially sample
order, are also dependent on more systematic choices unexplored in existing
works, such as number of classes, samples per class or choice of prompt format.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要