Target Category Agnostic Knowledge Distillation With Frequency-Domain Supervision

Existing knowledge distillation approaches require task-related data to train portable student networks for satisfactory performance. Nevertheless, in real-world applications, a majority of the data is unavailable over concerns about personal privacy, commercial confidentiality, etc. To mitigate the difficulty of acquiring the target dataset, the mainstream knowledge distillation methods generate training samples from the teacher network. However, the generated images still differ from the authentic ones, which limit the student network performance. To solve this issue, we propose a convenient and cost-negligible method to build datasets for training student networks. Specifically, we crawl the data from the web without knowing any category of the target dataset, named irrelevant category crawler data (ICCD). To prevent the performance collapse due to the data distribution gap between our ICCD and the target dataset, we propose a pseudo-classification strategy and frequency-domain supervision (PCFS) for target category agnostic knowledge distillation with ICCD. The pseudo-classification strategy classifies ICCD into different pseudo-categories by the teacher network, and uniformly but randomly samples images from each pseudo-category to construct the pseudo-target dataset. Furthermore, we transform the feature maps from the spatial domain to the frequency domain, and utilize the high- and low-frequency signals of the teacher network to impose strong constraints on the student network. Extensive experiments conducted on various test sets demonstrate the effectiveness of our proposed PCFS, which outperforms existing data-free methods and achieves comparable performance to those using the target training set. Code is available at https://github.com/SCUT-BIP-Lab/PCFS-DFKD.