Faster R–CNN–based apple detection in dense foliage fruiting wall trees using RGB and depth features for robotic harvesting

Apples in modern orchards with vertical-fruiting-wall trees are comparatively easier to harvest and specifically suitable for robotic picking, where accurate apple detection and obstacle-free access are fundamentally important. However, field images have complex backgrounds because of the presence of nontarget trees and fruit in adjacent rows. An outdoor machine vision system was developed with a low-cost Kinect V2 sensor to improve the accuracy of apple detection by filtering the background objects using depth features. A total of 800 set images were acquired in a commercial fruiting-wall Scifresh apple orchard with dense-foliage canopy. Images were collected in both daytime and nighttime with artificial light. The sensor was kept at 0.5 m to the tree canopies. A depth threshold of 1.2 m was used to remove background. Two Faster R–CNN based architectures ZFNet and VGG16 were employed to detect the Original-RGB and the Foreground-RGB images. Results showed that the highest average precision (AP) of 0.893 was achieved for the Foreground-RGB images with VGG16, which cost 0.181 s on average to process a 1920 × 1080 image. AP values for the Foreground-RGB images with ZFNet and VGG16 were both higher than those of the Original-RGB images. The results indicated that the use of a depth filter to remove background trees improved fruit detection accuracy by 2.5% and that only a minimal difference was found in processing speed between two image datasets. The proposed technique and results are expected to be applicable for robotic harvesting on fruiting-wall apple orchards.