MAE-MACD: the Masked Adversarial Contrastive Distillation Algorithm Grounded in Masked Autoencoders

In recent years, neural networks have been widely applied. However, adversarial attacks pose challenges to the secure deployment of neural networks. Adversarial training is one of the effective methods to train robust neural networks to resist such attacks. To address the high computational cost of adversarial training, we propose the masked adversarial contrastive distillation algorithm based on enhanced masked autoencoder (MAE-MACD). Unlike conventional approaches that demand the generation of adversarial samples in every iteration, MAE-MACD streamlines the process by requiring adversarial samples to be generated just once. First, MAE-MACD incorporates a feature learning module within the masked autoencoder, enabling the neural network to deduce global features from local ones. Second, it extracts the encoder and feature learning module from the masked autoencoder, utilizing them as the teacher model. In MACD, the knowledge distillation step involves training the model with different occlusion sizes and occlusion ratios, while the contrastive learning phase employs the same occlusion size but varying occlusion ratios to train the model. Finally, it fine-tunes the classification head using label information to ensure robust recognition performance. The experimental results demonstrate a significant improvement in neural network adversarial robustness achieved by MAE-MACD across CIFAR-10, CIFAR-100, and Tiny ImageNet, along with a reduced need for frequent adversarial sample generation.