Semantic granularity metric learning for visual search

Existing metric learning methods often do not consider different granularity in visual similarity. However, in many domains, images exhibit similarity at multiple granularities with visual semantic concepts, e.g.fashion demonstrates similarity ranging from clothing of the exact same instance to similar looks/design or common category. Therefore, training image triplets/pairs inherently possess different degree of information. Nevertheless, the existing methods often treat them with equal importance which hinder capturing underlying granularities in image similarity. In view of this, we propose a new semantic granularity metric learning (SGML) that develops a novel idea of detecting and leveraging attribute semantic space and integrating it into deep metric learning to capture multiple granularities of similarity. The proposed framework simultaneously learns image attributes and embeddings with multitask-CNN where the tasks are linked by semantic granularity similarity mapping to leverage correlations between the tasks. To this end, we propose a new soft-binomial deviance loss that effectively integrates informativeness of training samples into metric-learning on-the-fly during training. Compared to recent ensemble-based methods, SGML is conceptually elegant, computationally simple yet effective. Extensive experiments on benchmark datasets demonstrate its superiority e.g., 1–4.5%-Recall@1 improvement over the state-of-the-arts (Kim et al., 2018; Cakir et al., 2019) on DeepFashion-Inshop dataset.