Attention-based Feature Interaction for Efficient Online Knowledge Distillation

Existing online knowledge distillation (KD) methods solve the dependency problem of the high-capacity teacher model via mutual learning and ensemble learning. But they focus on the utilization of logits information in the last few layers and fail to construct a strong teacher model to better supervise student networks, leading to the inefficiency of KD. In this work, we propose a simple but effect...