Spark-SIFT: A Spark-Based Large-Scale Image Feature Extract System

The feature extraction is critical step in the image processing, with the popularity of the content-based image retrieval, how to extract the feature of the big-scale images quickly is become the very important and significant. In many big data dealing frameworks, spark is a memory based data processing framework with obvious advantages over processing speed. In this paper, we design a large-scale image feature extract framework based in spark. The framework contains three part,1) the base interface of image processing, 2) the sift algorithm in the spark. 3) The sequence of images. The problem of load unbalance will happened when the sizes of images to deal have wide difference, so to solve this problem, we propose the segmentation-image feature extract algorithm in the spark. In the algorithm, the big image is segmented to several parts for the more fast dealing speed. The experiment shows the framework has well speed compared with the single. When dealing the images which sizes is 4g in 7 machine, the speed reaches about 19.5. The segmentation-image feature extraction algorithm improves speed by 7.8 times when dealing 480M image set.