DeepKnowledge: Generalisation-Driven Deep Learning Testing
arxiv(2024)
摘要
Despite their unprecedented success, DNNs are notoriously fragile to small
shifts in data distribution, demanding effective testing techniques that can
assess their dependability. Despite recent advances in DNN testing, there is a
lack of systematic testing approaches that assess the DNN's capability to
generalise and operate comparably beyond data in their training distribution.
We address this gap with DeepKnowledge, a systematic testing methodology for
DNN-based systems founded on the theory of knowledge generalisation, which aims
to enhance DNN robustness and reduce the residual risk of 'black box' models.
Conforming to this theory, DeepKnowledge posits that core computational DNN
units, termed Transfer Knowledge neurons, can generalise under domain shift.
DeepKnowledge provides an objective confidence measurement on testing
activities of DNN given data distribution shifts and uses this information to
instrument a generalisation-informed test adequacy criterion to check the
transfer knowledge capacity of a test set. Our empirical evaluation of several
DNNs, across multiple datasets and state-of-the-art adversarial generation
techniques demonstrates the usefulness and effectiveness of DeepKnowledge and
its ability to support the engineering of more dependable DNNs. We report
improvements of up to 10 percentage points over state-of-the-art coverage
criteria for detecting adversarial attacks on several benchmarks, including
MNIST, SVHN, and CIFAR.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要