Test Case Prioritization for Deep Neural Networks

A variety of neural network-based models are increasingly being applied to real life, including face recognition, autonomous driving, etc. However, neural networks cannot fully guarantee reliability due to their black-box nature. Currently, a large number of test cases are usually used to ensure the safety of neural networks in different scenarios, and yet this will lead to high data labeling costs. Aiming at the above problems, we design a prioritized system for test cases based on Gini impurity. The system will model a series of dependent operations into a directed acyclic graph. After obtaining the inference result of the neural network model, the system computes the measurement value according to the formula of Gini impurity and uses this as the basis for comparison when sorting the test cases. The test case prioritization system can effectively filter out test cases that the neural network cannot make clear decisions, and help users inspect whether the data feature engineering is complete. In addition, using high-priority data for retraining can further improve the generalization ability of the model. The system has been deployed into MoocTest artificial intelligence infrastructure platform. Through a questionnaire survey, the satisfaction of our user experience reaches 92.8%.