Is Repeatability an Unbiased Criterion for Ranking Feature Detectors?

Most computer vision applications rely on algorithms finding local correspondences between different images. These algorithms detect and compare stable local invariant descriptors centered at scale-invariant keypoints. Because of the importance of the problem, new keypoint detectors and descriptors are constantly being proposed, each one claiming to perform better than the preceding ones. This raises the question of a fair comparison between very diverse methods. This evaluation has been based mainly on a repeatability criterion of the keypoints under a series of image perturbations (blur, illumination, noise, rotations, homotheties, homographies, etc). In this paper, we argue that the classic repeatability criterion is biased favoring algorithms producing redundant overlapped detections. To overcome this bias, we propose a variant of the repeatability rate taking into account the descriptors overlap. We apply this variant to revisit the popular benchmark by [Mikolajczyk et al. Int. J. Comput. Vis., 65 (2005), pp. 43-72], comparing several classic and recently introduced feature detectors. Experimental evidence shows that the hierarchy of these feature detectors is severely disrupted by the amended comparator.