Assignment Problem Based Deep Embedding

How to measure the similarity of samples is a fundamental problem in many computer vision tasks such as retrieval and clustering. Due to the rapid development of deep neural networks, deep metric learning has been widely studied. Some studies focus on the hard sample mining strategy for triplet loss. We observe that hard mining strategies are also vital for contrastive loss. But the hardest mining strategy for contrastive loss is sensitive to outliers. In this paper, based on combinatorial information of sample pairs, we propose a novel linear assignment problem based hard sample mining strategy for contrastive loss to learn feature embeddings. Specifically, our method can assign 0/1 weight to sample pairs for the hard sample selection by maximizing a linear assignment loss and ensure that each sample is only included by one pair for the optimization. Our method can obtain the state-of-the-art performance on the CUB-200-2011, Cars196, and In-shop datasets with the GoogLeNet network.