LabelMe: Online Image Annotation and Applications

Abstract—Central to the development,of computer,vision systems is the collection and use of annotated images spanning our visual world. Annotations may,include information about the identity, spatial extent, and viewpoint of the objects p resent in a depicted scene. Such a database is useful for the training and evaluation of computer vision systems. Motivated by the availability of images on the internet, we introduced a web-based annotation tool that allows online users to label objects and their spatial extent in images. To date, we have collected over 400K annotations that span a variety of different scene and object classes. In this paper, we show the contents of the database, its growth over time, and statistics of its usage. In addition, we explore and survey applications of the database in the areas of computer vision and computer graphics. Particularly, we show how,to extract the real-world 3D coordinates of images in a variety of scenes using only the user-provided object annotations. The output 3D information is comparable to the quality produced by a laser range scanner. We also characterize the space of the images in the database by analyzing (i) statistics of the co- occurrence of large objects in the images and (ii) the spatial layout of the labeled images. Index Terms—online annotation tool, image database, object