An Experimental Study of DNN Operator-Level Performance on Edge Devices

Deep Neural Network (DNN) has been widely used on edge devices with the development of Internet of Things (IoT). However, the limited resources of edge devices affect the deployment of DNN model on edge. Some works devote to deploying DNN models to edge devices have great demand on DNN inference latency prediction, but the typical prediction methods usually lack the knowledge of operator-level performance at DNN runtime, affecting the accuracy and generality. In this paper, we conduct a comprehensive experimental study of DNN operator-level performance on edge devices. We measured the inference latency performance of models and operator pairs in TensorRT and PyTorch frameworks on a Jetson Xavier NX Developer kit and a Jetson Nano Developer kit. Based on the experimental results, we summarize some useful hidden rules about inference latency relation between separated operators and operator blocks as well as operator fusion in different deep learning frameworks. Furthermore, we propose several implementation tips in aid of designing DNN inference latency predictor with higher accuracy and less complexity.