Operation is the hardest teacher: estimating DNN accuracy looking for mispredictions
arXiv (Cornell University)(2021)
摘要
Deep Neural Networks (DNN) are typically tested for accuracy relying on a set
of unlabelled real world data (operational dataset), from which a subset is
selected, manually labelled and used as test suite. This subset is required to
be small (due to manual labelling cost) yet to faithfully represent the
operational context, with the resulting test suite containing roughly the same
proportion of examples causing misprediction (i.e., failing test cases) as the
operational dataset. However, while testing to estimate accuracy, it is
desirable to also learn as much as possible from the failing tests in the
operational dataset, since they inform about possible bugs of the DNN. A smart
sampling strategy may allow to intentionally include in the test suite many
examples causing misprediction, thus providing this way more valuable inputs
for DNN improvement while preserving the ability to get trustworthy unbiased
estimates. This paper presents a test selection technique (DeepEST) that
actively looks for failing test cases in the operational dataset of a DNN, with
the goal of assessing the DNN expected accuracy by a small and ”informative”
test suite (namely with a high number of mispredictions) for subsequent DNN
improvement. Experiments with five subjects, combining four DNN models and
three datasets, are described. The results show that DeepEST provides DNN
accuracy estimates with precision close to (and often better than) those of
existing sampling-based DNN testing techniques, while detecting from 5 to 30
times more mispredictions, with the same test suite size.
更多查看译文
关键词
dnn accuracy,hardest teacher
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要