A Computational Study on Calibrated VGG19 for Multimodal Learning and Representation in Surveillance

This research discusses the pre-trained deep learning architecture for the multimodal learning and representation in surveillance system. This framework generates a single image from the integration of the multi sensor information, which includes the infrared and visible. We use visible and infrared as the information in different spectrum of light, in term of contrast ratio and visibility. We start with image registration to align coordinates so the decomposition of the source image into the sub bands is possible. The VGG-19 and the weighted averaging are utilized for the feature extraction and transfer learning task. This is conducted thorough empirical research by implementing a series of methodology studies to evaluate how pre-trained deep learning techniques enhance overall fusion performance and improve recognition and detection capability. This study also contains a comparison of the performance of spatial and frequency algorithms in contrast to the deep learning based method for the surveillance system. The research work is concluded by evaluating the performance measure of the proposed fusion algorithm with the traditional algorithm.