Symbiotic Artificial Intelligence: Order Picking And Ambient Sensing

Using egocentric video and head motion data from 67 order picking tasks (244 picks;149 orders), we learn visual models of the 10 objects picked to fulfill the orders. Boundary segmentations of the four actions (pick, carry, place, carry empty) of order picking had an average test RMSE of 1.11 seconds using computer vision and 5.53 seconds using only head motion $( \approx 39.8$ seconds/task). The 10 objects were clustered with 93.8% accuracy using weak supervision provided by the picks (which could occur in any order) specified in the tasks. We apply the 10 resulting models on independent test data to recognize three objects involving 50 tasks (185 picks;98 orders) and 10 objects involving 10 tasks (35 picks;24 orders). Accuracy was up to 90.3% and 69.1%, respectively. We propose order picking as a practical use case of egocentric Symbiotic AI, where ambient sensing is used without explicit supervision to train an agent which can then help the user improve task speed and accuracy. ¹