Distributionally Robust Deep Learning as a Generalization of Adversarial Training

Machine learning models are vulnerable to adversarial attacks at test time: a correctly classified test example can be slightly perturbed to cause a misclassification. Training models that are robust to these attacks, and theoretical understanding of such defenses are active research areas. Adversarial Training (AT) via robust optimization is a promising approach, where the model is trained against an adversary acting on the training set, but it is less clear how to reason about perturbations on the unseen test set. Distributionally Robust Optimization (DRO) with Wasserstein distance is an interesting theoretical tool for understanding robustness and generalization, but it has been limited algorithmically to simple models. We link DRO and AT both theoretically and algorithmically: AT is a special case of DRO, and in general DRO yields a stronger adversary. We also give an algorithm for DRO for neural networks that is no more expensive than AT.