Deep Passenger State Monitoring Using Viewpoint Warping

The advent of autonomous and semi-autonomous vehicles has meant passengers now play a more significant role in the safety and comfort of vehicle journeys. In this paper, we propose a deep learning method to monitor and classify passenger state with camera data. The training of a convolutional neural network is supplemented by data captured from vehicle occupants in different seats and from different viewpoints. Existing driver data or data from one vehicle is augmented by viewpoint warping using planar homography, which does not require knowledge of the source camera parameters, and overcomes the need to re-train the model with large amounts of additional data. To analyse the performance of our approach, data is collected on occupants in two different vehicles, from different viewpoints inside the vehicle. We show that the inclusion of the additional training data and augmentation by homography increases the average passenger state classification rate by 11.1%. We conclude by proposing how occupant state may be used holistically for activity recognition and intention prediction for intelligent vehicle features.