TU-Net and TDeepLab: Deep Learning-Based Terrain Classification Robust to Illumination Changes, Combining Visible and Thermal Imagery

In this paper we propose two novel deep learning-based terrain classification methods robust to illumination changes. The use of cameras is challenged by a variety of factors, of most importance being the changes in illumination. On the other hand, since the temperature of various types of terrains depends on the thermal characteristics of the terrain, the terrain classification can be aided by utilizing the thermal information in addition to visible information. Thus we propose 'TU-Net (Two U-Net)' based on the U-Net and 'TDeepLab (Two DeepLab)' based on DeepLab, which combine visible and thermal images and train the network robust to illumination changes implicitly. To improve the network's learning capability, we expand the proposed methods to the Siamese-based method, which explicitly trains the network to be robust to illumination changes. We also investigate multiple options to fuse the visible and thermal images at at the bottom layer, middle layer, or the top layer of the network. We evaluate the proposed methods with a challenging new dataset consisting of visible and thermal images, which were collected from 10 am till 5 pm (after sunset), and we show the effectiveness of the proposed methods.