Advancing Deep Active Learning Data Subset Selection: Unifying Principles with Information-Theory Intuitions
CoRR(2024)
摘要
At its core, this thesis aims to enhance the practicality of deep learning by
improving the label and training efficiency of deep learning models. To this
end, we investigate data subset selection techniques, specifically active
learning and active sampling, grounded in information-theoretic principles.
Active learning improves label efficiency, while active sampling enhances
training efficiency. Supervised deep learning models often require extensive
training with labeled data. Label acquisition can be expensive and
time-consuming, and training large models is resource-intensive, hindering the
adoption outside academic research and “big tech.” Existing methods for data
subset selection in deep learning often rely on heuristics or lack a principled
information-theoretic foundation. In contrast, this thesis examines several
objectives for data subset selection and their applications within deep
learning, striving for a more principled approach inspired by information
theory. We begin by disentangling epistemic and aleatoric uncertainty in single
forward-pass deep neural networks, which provides helpful intuitions and
insights into different forms of uncertainty and their relevance for data
subset selection. We then propose and investigate various approaches for active
learning and data subset selection in (Bayesian) deep learning. Finally, we
relate various existing and proposed approaches to approximations of
information quantities in weight or prediction space. Underpinning this work is
a principled and practical notation for information-theoretic quantities that
includes both random variables and observed outcomes. This thesis demonstrates
the benefits of working from a unified perspective and highlights the potential
impact of our contributions to the practical application of deep learning.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要