A CNN-based multi-task framework for weather recognition with multi-scale weather cues

Automatic image-based weather recognition has a great significance in the practical use of computer visual applications. The weather recognition approaches via multi-task learning obtain the description of weather conditions by jointly tackling weather-cue segmentation task and weather classification task. However, the existing approaches neglect the multiple scales of weather cues, which may decrease the ability to deal with both large and small weather cues in weather-cue segmentation. They also fail to assign appropriate weights, which the multi-task learning for weather recognition relies on, to the two tasks. To address these problems, we propose a novel end-to-end Convolutional Neural Network (CNN)-based Multi-task Weather Recognition Network with multi-scale weather cues (CMWRN). Specifically, in the weather-cue segmentation task, we propose to segment weather cues, e.g. blue sky and black clouds, by capturing multi-scale weather-cue features. Arbitrary-scale weather-cue regions in the outdoor images can be accurately and effectively classified via resampling the weather-cue features at multiple scales. In the weather classification task, a generalized pooling method is introduced to generate more discriminative weather representations. To further promote the weather classification, the multi-scale weather-cue features are exploited to provide weather clues for weather representations. Moreover, we develop an adaptive weighting scheme to automatically balance the weather classification task and weather-cue segmentation task for more effective training. Compared with the state-of-the-arts, experimental results on two public available benchmark datasets demonstrate the superior performance of our approach.