Vision-based automatic order check method for online medicine dispensing cabinet under incomplete data

In recent years, computer vision technology has been increasingly used in the retail industry, especially in unmanned retail. In the online medicine dispensing cabinet where the order images are captured along with the customer’s online order information, an automatic order check method is needed to verify whether the picked items match the order information and avoid the mis-dispensing of medicines. Well-built images dataset and annotations are needed for the task. However, it is difficult to construct datasets with fine annotations and keep the images up-to-date in the application scenario. Automatic order check is facing the problems of small samples, diverse and dynamic categories, weak supervision, and domain gaps. To deal with the order check task under incomplete data in real scenarios, we carried out the following works: (1) We proposed a framework for automatic order check (AOC), and collected an incomplete dataset in natural scenes. (2) We proposed a foreground detection method based on conventional image processing. Image matching methods based on local descriptors and text recognition were designed for classification. (3) We proposed a data simulation pipeline for rotated object detection model training with weak supervision. Our methods showed promising results on our challenging dataset.