Rain streaks removal from single image based on texture constraint of background scene

•We propose a rain streaks removal method in single image based on texture constraint of background scene, which is established by explicitly utilizing the frequency, direction characteristic of rain streaks, and the non-rain texture clues learned from natural images.•Different from adopting an online learning program to provide constraints, we propose a ICA-based fusion method. The ICA allows to learn once for further use, without the need to learn for each rain image. It reduces the computation time.•A new decomposition model is proposed, where a texture constraint for background scene and a low-rank appearance constraint for rain streaks are both utilized to reduce the confusion between image texture and rain structure.