Learning Early Exit for Deep Neural Network Inference on Mobile Devices through Multi-Armed Bandits

We present a novel learning framework that utilizes the early exit of Deep Neural Network (DNN), a device-only solution that reduces the latency of inference by sacrificing a reasonable degree of accuracy. Choosing the optimal exit point is challenging as the delay and the accuracy of each exit point are random and cannot be known in advance. The problem is further complicated as the overall duration of the processing is also unknown. To this end, we propose Learning Early Exit (LEE), an online learning scheme based on multi-armed bandits analysis. LEE efficiently learns the optimal exit point for mobile-based DNN inference while simultaneously balancing the exploration-exploitation trade-off. LEE differs from the standard bandit analyses in two ways: the reward of choosing each exit point addresses the confidence-latency trade-off, and the time duration between each action is random (i.e., the latency of each action is random). LEE addresses the aforementioned challenges and it achieves asymptotically optimal performance. We implement a real-world system with a real-time testbed that can be deployed in a driving system. DNN models with multiple exit points are trained and deployed in the testbed so that the performance of LEE and benchmark schemes can be tested and compared. The result denotes that LEE substantially outperforms the benchmark schemes.