# Provably Robust Metric Learning

NeurIPS 2020, 2020.

Keywords:

Weibo:

Abstract:

Metric learning is an important family of algorithms for classification and similarity search, but the robustness of learned metrics against small adversarial perturbations is less studied. In this paper, we show that existing metric learning algorithms, which focus on boosting the clean accuracy, can result in metrics that are less rob...More

Code:

Data:

Introduction

- Metric learning has been an important family of machine learning algorithms and has achieved successes on several problems, including computer vision [24, 17, 18], text analysis [27], meta learning [38, 35] and others [34, 45, 47].
- Given a set of training samples, metric learning aims to learn a good distance measurement such that items in the same class are closer to each other in the learned metric space, which is crucial for classification and similarity search
- Since this objective is directly related to the assumption of nearest neighbor classifiers, most of the metric learning algorithms can be naturally and successfully combined with K-Nearest Neighbor (K-NN) classifiers.
- None of these previous methods are trying to find a metric that is robust to small input perturbations

Highlights

- Metric learning has been an important family of machine learning algorithms and has achieved successes on several problems, including computer vision [24, 17, 18], text analysis [27], meta learning [38, 35] and others [34, 45, 47]
- Recent research in adversarial defense of neural networks has shifted to the concept of “certified defense”, where the defender needs to provide a certification that no adversarial examples exist within a certain input region [43, 11, 48]
- Computing the minimal adversarial perturbation is intractable for K-Nearest Neighbor (K-NN), so to make the problem solvable, we propose an efficient formulation for lower-bounding the minimal adversarial perturbation, and this lower bound can be represented as an explicit function of M to enable the gradient computation
- Metric learning for nearest neighbor classifiers A nearest-neighbor classifier based on a Mahalanobis distance could be characterized by a training dataset and a positive semi-definite matrix
- We propose a novel metric learning method named adversarially robust metric learning (ARML) to obtain a robust Mahalanobis distance that can be robust to adversarial input perturbations
- Experiments show that the proposed method can improve both clean errors and robust errors compared with existing metric learning algorithms

Conclusion

- The authors propose a novel metric learning method named ARML to obtain a robust Mahalanobis distance that can be robust to adversarial input perturbations.
- Experiments show that the proposed method can improve both clean errors and robust errors compared with existing metric learning algorithms

Summary

## Introduction:

Metric learning has been an important family of machine learning algorithms and has achieved successes on several problems, including computer vision [24, 17, 18], text analysis [27], meta learning [38, 35] and others [34, 45, 47].- Given a set of training samples, metric learning aims to learn a good distance measurement such that items in the same class are closer to each other in the learned metric space, which is crucial for classification and similarity search
- Since this objective is directly related to the assumption of nearest neighbor classifiers, most of the metric learning algorithms can be naturally and successfully combined with K-Nearest Neighbor (K-NN) classifiers.
- None of these previous methods are trying to find a metric that is robust to small input perturbations
## Conclusion:

The authors propose a novel metric learning method named ARML to obtain a robust Mahalanobis distance that can be robust to adversarial input perturbations.- Experiments show that the proposed method can improve both clean errors and robust errors compared with existing metric learning algorithms

- Table1: Certified robust errors of Mahalanobis 1-NN. The best (minimum) certified robust errors among all methods are in bold. Note that the certified robust errors of 1-NN are also the optimal attack errors
- Table2: Certified robust errors (left) and empirical robust errors (right) of Mahalanobis K-NN. The best (minimum) robust errors among all methods are in bold
- Table3: Dataset statisitcs

Related work

- Metric Learning Metric learning aims to learn a new distance using supervision concerning the learned distance [23]. In this paper, we mainly focus on the linear metric learning: the learned distance is the squared Euclidean distance after applying the transformation G globally, i.e., the

Mahalanobis distance [15, 12, 41, 20, 36]. There are also nonlinear models for metric learning, such as kernelized metric learning [25, 6], local metric learning [14, 40] and deep metric learning [10, 32]. Robustness verification for nonlinear metric learning and learning a provably robust non-linear metric would be an interesting future work.

Adversarial robustness of neural networks Empirical defense aims to learn a classifier which is robust to some adversarial attacks [26, 29], but has no guarantee for the robustness to other stronger (or unknown) adversarial attacks [4, 1]. In contrast, certified defense provides a guarantee that no adversarial examples exist within a certain input region [43, 11, 48]. The basic idea of these certified defense methods is to minimize the certified robust training error. However, all these methods for neural networks rely on the assumption of smoothness of the classifier, and hence could not be applied to the nearest neighbor classifiers.

Reference

- A. Athalye, N. Carlini, and D. Wagner. Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples. In International Conference on Machine Learning (ICML), pages 274–283, 2018.
- B. Biggio and F. Roli. Wild patterns: Ten years after the rise of adversarial machine learning. Pattern Recognition, 84:317–331, 2018.
- W. Brendel, J. Rauber, and M. Bethge. Decision-based adversarial attacks: Reliable attacks against black-box machine learning models. In International Conference on Learning Representations (ICLR), 2018.
- N. Carlini and D. Wagner. Towards evaluating the robustness of neural networks. In IEEE Symposium on Security and Privacy (SP), pages 39–57, 2017.
- C.-C. Chang and C.-J. Lin. LIBSVM: A library for support vector machines. ACM Transactions on Intelligent Systems and Technology, 2(3):27, 2011.
- R. Chatpatanasiri, T. Korsrilabutr, P. Tangchanachaianan, and B. Kijsirikul. A new kernelization framework for mahalanobis distance learning algorithms. Neurocomputing, 73(10):1570–1579, 2010.
- P.-Y. Chen, H. Zhang, Y. Sharma, J. Yi, and C.-J. Hsieh. ZOO: Zeroth order optimization based black-box attacks to deep neural networks without training substitute models. In ACM Conference on Computer and Communications Security (CCS) Workshop on Artificial Intelligence and Security (AISec), pages 15–26, 2017.
- M. Cheng, T. M. Le, P.-Y. Chen, H. Zhang, J. Yi, and C.-J. Hsieh. Query-efficient hard-label black-box attack: An optimization-based approach. In International Conference on Learning Representations (ICLR), 2019.
- M. Cheng, S. Singh, P.-Y. Chen, S. Liu, and C.-J. Hsieh. Sign-opt: A query-efficient hard-label adversarial attack. In International Conference on Learning Representations (ICLR), 2020.
- S. Chopra, R. Hadsell, and Y. LeCun. Learning a similarity metric discriminatively, with application to face verification. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), volume 1, pages 539–546, 2005.
- J. Cohen, E. Rosenfeld, and Z. Kolter. Certified adversarial robustness via randomized smoothing. In International Conference on Machine Learning (ICML), pages 1310–1320, 2019.
- J. V. Davis, B. Kulis, P. Jain, S. Sra, and I. S. Dhillon. Information-theoretic metric learning. In International Conference on Machine learning (ICML), pages 209–216, 2007.
- W. de Vazelhes, C. Carey, Y. Tang, N. Vauquier, and A. Bellet. metric-learn: Metric learning algorithms in python. CoRR preprint arXiv:1908.04710, arXiv/1908.04710, 2019.
- A. Frome, Y. Singer, F. Sha, and J. Malik. Learning globally-consistent local distance functions for shape-based image retrieval and classification. In International Conference on Computer Vision (ICCV), pages 1–8, 2007.
- J. Goldberger, G. E. Hinton, S. T. Roweis, and R. R. Salakhutdinov. Neighbourhood components analysis. In Advances in Neural Information Processing Systems (NeurIPS), pages 513–520, 2004.
- I. J. Goodfellow, J. Shlens, and C. Szegedy. Explaining and harnessing adversarial examples. In International Conference on Learning Representations (ICLR), 2015.
- M. Guillaumin, J. Verbeek, and C. Schmid. Is that you? metric learning approaches for face identification. In International Conference on Computer Vision (ICCV), pages 498–505, 2009.
- G. Hua, M. Brown, and S. Winder. Discriminant embedding for local image descriptors. In International Conference on Computer Vision (ICCV), pages 1–8, 2007.
- A. Ilyas, L. Engstrom, and A. Madry. Prior convictions: Black-box adversarial attacks with bandits and priors. In International Conference on Learning Representations (ICLR), 2019.
- P. Jain, B. Kulis, and I. S. Dhillon. Inductive regularized learning of kernel functions. In Advances in Neural Information Processing Systems (NeurIPS), pages 946–954, 2010.
- D. P. Kingma and J. L. Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015.
- M. K. Kozlov, S. P. Tarasov, and L. G. Khachiyan. The polynomial solvability of convex quadratic programming. USSR Computational Mathematics and Mathematical Physics, 20(5):223–228, 1980.
- B. Kulis. Metric learning: A survey. Foundations and Trends in Machine Learning, 5(4):287– 364, 2013.
- B. Kulis, P. Jain, and K. Grauman. Fast similarity search for learned metrics. IEEE Transactions on Pattern Analysis and Machine Intelligence, 31(12):2143–2157, 2009.
- B. Kulis, M. Sustik, and I. Dhillon. Learning low-rank kernel matrices. In International Conference on Machine learning (ICML), pages 505–512, 2006.
- A. Kurakin, I. J. Goodfellow, and S. Bengio. Adversarial machine learning at scale. In International Conference on Learning Representations (ICLR), 2017.
- G. Lebanon. Metric learning for text documents. IEEE Transactions on Pattern Analysis and Machine Intelligence, 28(4):497–508, 2006.
- Y. LeCun, L. Bottou, Y. Bengio, P. Haffner, et al. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998.
- A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations (ICLR), 2018.
- M. Mirman, T. Gehr, and M. Vechev. Differentiable abstract interpretation for provably robust neural networks. In International Conference on Machine Learning (ICML), pages 3578–3586, 2018.
- N. Papernot, P. D. McDaniel, and I. J. Goodfellow. Transferability in machine learning: from phenomena to black-box attacks using adversarial samples. CoRR, abs/1605.07277, 2016.
- F. Schroff, D. Kalenichenko, and J. Philbin. Facenet: A unified embedding for face recognition and clustering. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 815–823, 2015.
- C. Sitawarin and D. A. Wagner. Minimum-norm adversarial examples on KNN and KNN-based models. CoRR, arXiv/2003.06559, 2020.
- M. Slaney, K. Q. Weinberger, and W. White. Learning a metric for music similarity. In International Symposium/Conference on Music Information Retrieval, pages 313–318, 2008.
- J. Snell, K. Swersky, and R. Zemel. Prototypical networks for few-shot learning. In Advances in Neural Information Processing Systems (NeurIPS), pages 4077–4087, 2017.
- M. Sugiyama. Dimensionality reduction of multimodal labeled data by local fisher discriminant analysis. Journal of Machine Learning Research (JMLR), 8:1027–1061, 2007.
- C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. J. Goodfellow, and R. Fergus. Intriguing properties of neural networks. In International Conference on Learning Representations (ICLR), 2014.
- O. Vinyals, C. Blundell, T. Lillicrap, K. Kavukcuoglu, and D. Wierstra. Matching networks for one shot learning. In Advances in Neural Information Processing Systems (NeurIPS), pages 3637–3645, 2016.
- L. Wang, X. Liu, J. Yi, Z.-H. Zhou, and C.-J. Hsieh. Evaluating the robustness of nearest neighbor classifiers: A primal-dual perspective. CoRR, abs/1906.03972, 2019.
- K. Q. Weinberger and L. K. Saul. Fast solvers and efficient implementations for distance metric learning. In International Conference on Machine learning (ICML), pages 1160–1167, 2008.
- K. Q. Weinberger and L. K. Saul. Distance metric learning for large margin nearest neighbor classification. Journal of Machine Learning Research (JMLR), 10:207–244, 2009.
- L. Weng, H. Zhang, H. Chen, Z. Song, C.-J. Hsieh, L. Daniel, D. Boning, and I. Dhillon. Towards fast computation of certified robustness for relu networks. In International Conference on Machine Learning (ICML), pages 5276–5285, 2018.
- E. Wong and Z. Kolter. Provable defenses against adversarial examples via the convex outer adversarial polytope. In International Conference on Machine Learning (ICML), pages 5283– 5292, 2018.
- H. Xiao, K. Rasul, and R. Vollgraf. Fashion-MNIST: a novel image dataset for benchmarking machine learning algorithms. CoRR, abs/1708.07747, 2017.
- H. Xiong and X.-w. Chen. Kernel-based distance metric learning for microarray data classification. BMC bioinformatics, 7(1):299, 2006.
- Y.-Y. Yang, C. Rashtchian, Y. Wang, and K. Chaudhuri. Robustness for non-parametric classification: A generic attack and defense. In International Conference on Artificial Intelligence and Statistics (AISTATS), 2020.
- H.-J. Ye, D.-C. Zhan, X.-M. Si, Y. Jiang, and Z.-H. Zhou. What makes objects similar: A unified multi-metric learning approach. In Advances in Neural Information Processing Systems (NeurIPS), pages 1235–1243, 2016.
- H. Zhang, H. Chen, C. Xiao, S. Gowal, R. Stanforth, B. Li, D. Boning, and C.-J. Hsieh. Towards stable and efficient training of verifiably robust neural networks. In International Conference on Learning Representations (ICLR), 2020.

Tags

Comments