Integrated Vision-Body Sensing System for Tracking People in Intelligent Environments

An integrated sensing architecture and “big data” analytics that leverages real-time human biometric monitoring with state-of-the-art human activity recognition (HAR) can open exciting new possibilities in human health and performance monitoring. We present a framework for integrating data streams from computer vision detection algorithms and on-body sensors through a series of experiments on human subjects with wearable sensors. These experiments were carried out in the MCity facility at the University of Michigan. Our tests utilize urban environments of Mcity to obtain human activity data streams from a network of accessible cameras and wearable biometric sensors transmitted via LoRa to an access point. Data are then forwarded to Amazon’s cloud computing platform, Amazon Web Services (AWS). Once on AWS, GPU servers running real-time HAR algorithms can access and integrate biometric sensor data. We show the results of computer vision tool development using image/video based object detection, tracking, and mapping. Our work of developing real-time on-body device combined with device-free activities recognition will advance capabilities of monitoring personnel in their environments for variety of industrial and defense applications.