Large-scale visual search and similarity for e-commerce

Visual search and similarity can aid an e-commerce platform by providing appropriate recommendations where semantic labeling and associated metadata does not always exist. In this work, we detail the specifics of our system that powers visually similar recommendations. While a common approach leverages learned representation from common classification tasks using DNN’s, the crux of the problem are the labels and ontology that is applied in order for the DNN to functionally perform. Our proposed approach in production for a variety of products is to supply these recommendations based on a defined taxonomy through a hierarchy that has been carefully curated while additionally scaled up through our platform's natural crowd-sourcing interface. To scale the use of these taxonomies in production, we quantization schemes for retrieving approximate nearest neighbors after applying base transformations on the images using Apache Beam and Tensorflow Transform. The nearest neighbor retrievals are based on using a ResNet model architecture, trained from scratch on 3000+ classes. These are trained daily in a distributed fashion and optimizing data throughput. Finally, in order to verify the appropriateness, we use an extensive human evaluation pipeline and quality control. In this work, we share our product design learnings from the various attempts/experiments we conducted for a successful launch.