A Practical Guide to CNNs and Fisher Vectors for Image Instance Retrieval

With deep learning becoming the dominant approach in computer vision, the use of representations extracted from Convolutional Neural Nets (CNNs) is quickly gaining ground on Fisher Vectors (FVs) as favoured state-of-the-art global image descriptors for image instance retrieval. While the good performance of CNNs for image classification are unambiguously recognised, which of the two has the upper hand in the image retrieval context is not entirely clear yet.We propose a comprehensive study that systematically evaluates FVs and CNNs for image instance retrieval. The first part compares the performances of FVs and CNNs on multiple publicly available data sets and for multiple criteria. We show that no descriptor is systematically better than the other and that performance gains can usually be obtained by using both types together. The second part of the study focuses on the impact of geometrical transformations. We show that performance of CNNs can quickly degrade in the presence of certain transformations and propose a number of ways to incorporate the required invariances in the CNN pipeline.Our findings are organised as a reference guide offering practically useful and simply implementable guidelines to anyone looking for state-of-the-art global descriptors best suited to their specific image instance retrieval problem. HighlightsCNNs exhibit very limited invariance to rotation changes compared to FVDoG.CNNs are more robust to scale changes than any variants of FV.Max-pooling across rotated/scaled database images gains rotation/scale invariance.Combining FV with CNN can improve retrieval accuracy by a significant margin.