Adversarial Deep Mutual Learning

Recently more attention has been paid to neural network compression due to the increasing requirements about limit memory with high performance. The typical methods based on knowledge distillation have to fix a pre-trained teacher model, and most result-orient methods are limited by the teacher model. In this work, we propose a novel lightweight network training method, which apply adversarial learning to deep mutual learning and train multiple networks in just one framework without pre-trained. We focus on the potential relationships among all networks in our framework to form the ingenious loss rather than these manual designed. Extensive experimental results on several datasets illustrate that the proposed method can significantly outperform other state-of-the-art methods.