Towards Intelligent Fruit Picking with In-hand Sensing

Studies have shown that picking techniques play an important role in determining fruit quality at harvest (e.g. bruising, stem retention, etc). When picking fruit such as apples and pears, professional pickers use active perception, incorporating both visual and tactile input about fruit orientation, stem location, and the fruit's immediate surroundings. This combination of tactile, visual, and force feedback is what enables human workers to execute dynamic movements that quickly and efficiently remove fruit from the tree without damage. However, much of the prior work on robotic fruit picking has formulated the harvesting problem as a position-control problem, using visual feedback for closed-loop end-effector placement while disregarding feedback on physical contact. As a first step towards more intelligent fruit picking - combining proprioception, localized sensing, and observed forces- we have developed a custom end-effector with multiple in-hand sensors, including tactile sensors on the fingertips. This paper presents the mechatronic design of the device as well as results from multiple outdoor picking trials with a Honeycrisp apple tree. Preliminary results show that, with multi-modal sensing, fruit slip, fruit separation from the tree, and fruit release from the hand can be detected.