Multi-modal Visual Withdrawal Detection for Inventory Management on a Robotic Care Cart

We present a camera-based withdrawal detection system for inventory management on a robotic care cart. Navigating to the storage room for refilling the items takes time and energy for both robotic and manual care carts. Hence, tracking the inventory is crucial for efficiency. Currently, this task is performed by the nurses or care staff. Our approach aims at reducing the workload of the staff by warning them if the stock of an item is critically low. The extendable modular architecture combines different visual modalities such as hand tracking, rgb- and depth-based changes in drawer partitions and a motion cue, by fusing them for higher robustness. Our experimental results support the need for a detection system for such an application. Although the hand tracking modality provides the best accuracy among the other modalities, the fused result from different modalities outperforms regarding robustness.