Image De-raining Using a Conditional Generative Adversarial Network

    He Zhang
    He Zhang
    Vishwanath Sindagi
    Vishwanath Sindagi

    IEEE Transactions on Circuits and Systems for Video Technology, 2017.

    Cited by: 306|Bibtex|Views18|Links
    EI
    Keywords:
    rainy imagehigh frequencyprior informationimage decompositionDeep Convolutional GANsMore(16+)
    Wei bo:
    We investigate a new point of view in addressing the single image de-raining problem

    Abstract:

    Severe weather conditions such as rain and snow adversely affect the visual quality of images captured under such conditions thus rendering them useless for further usage and sharing. In addition, such degraded images drastically affect performance of vision systems. Hence, it is important to solve the problem of single image de-raining/d...More

    Code:

    Data:

    0
    Introduction
    • It has been widely acknowledged that unpredictable impairments such as illumination, noise and severe weather conditions adversely affect the performance of many computer vision algorithms such as detection, classification and tracking.
    • Sparse coding-based clustering method [8] is among the first ones to tackle the single image de-raining problem where the authors proposed to solve it in the image decomposition framework.
    • They first separated the input image into low frequency and high frequency images using a bilateral filter.
    • Due to the same assumption, their method generates artifacts around the rain-streak components in the resulting images
    Highlights
    • It has been widely acknowledged that unpredictable impairments such as illumination, noise and severe weather conditions adversely affect the performance of many computer vision algorithms such as detection, classification and tracking
    • We investigate conditional generative adversarial networks (GANs) to address this issue, where a pre-trained discriminator network is used as a guide to synthesize images free from weather-based degradations
    • We proposed a conditional generative adversarial networks-based algorithm for the removal of rain streaks form a single image
    • In comparison to the existing approaches which attempt to solve the de-raining problem in an image decomposition framework by using prior information, we investigated the use of conditional generative adversarial networks for synthesizing de-rained image from a given input rainy image
    • For improved stability in training and reducing artifacts introduced by generative adversarial networks in the output images, we propose the use of a new refined loss function in the generative adversarial networks optimization framework
    • The proposed Image De-raining Conditional General Adversarial Network method is compared against baseline configurations to illustrate the performance gains obtained by introducing the refined perceptual loss into the conditional generative adversarial networks framework
    Methods
    • The discriminator sub-network D, as shown in the bottom part in Figure 3, serves to distinguish ‘fake’ de-rained image synthesized by the generators from corresponding ground truth ‘real’ image.
    • It can be viewed as a guidance for the generator G.
    • In the following sub-sections, the authors discuss these important parts in detail starting with GAN objective function followed by generator/discriminator subnetworks and refined perceptual loss
    Results
    • The authors present details of the experiments and quality measures used to evaluate the proposed ID-CGAN method.
    • 1) Synthetic dataset: Due to the lack of availability of large size datasets for training and evaluation of single image de-raining, the authors synthesized a new set of training and testing samples in the experiments.
    • The training set consists of a total of 700 images, where 500 images are randomly chosen from the first 800 images in the UCID dataset [45] and 200 images are randomly chosen from the BSD-500’s training set [46].
    Conclusion
    • The authors proposed a conditional GAN-based algorithm for the removal of rain streaks form a single image.
    • In comparison to the existing approaches which attempt to solve the de-raining problem in an image decomposition framework by using prior information, the authors investigated the use of conditional GANs for synthesizing de-rained image from a given input rainy image.
    • The proposed ID-CGAN method is compared against baseline configurations to illustrate the performance gains obtained by introducing the refined perceptual loss into the conditional GAN framework.
    • The authors aim to build upon the conditional GAN framework to overcome these drawback and investigate the possibility of using similar structures for solving related problems
    Summary
    • Introduction:

      It has been widely acknowledged that unpredictable impairments such as illumination, noise and severe weather conditions adversely affect the performance of many computer vision algorithms such as detection, classification and tracking.
    • Sparse coding-based clustering method [8] is among the first ones to tackle the single image de-raining problem where the authors proposed to solve it in the image decomposition framework.
    • They first separated the input image into low frequency and high frequency images using a bilateral filter.
    • Due to the same assumption, their method generates artifacts around the rain-streak components in the resulting images
    • Methods:

      The discriminator sub-network D, as shown in the bottom part in Figure 3, serves to distinguish ‘fake’ de-rained image synthesized by the generators from corresponding ground truth ‘real’ image.
    • It can be viewed as a guidance for the generator G.
    • In the following sub-sections, the authors discuss these important parts in detail starting with GAN objective function followed by generator/discriminator subnetworks and refined perceptual loss
    • Results:

      The authors present details of the experiments and quality measures used to evaluate the proposed ID-CGAN method.
    • 1) Synthetic dataset: Due to the lack of availability of large size datasets for training and evaluation of single image de-raining, the authors synthesized a new set of training and testing samples in the experiments.
    • The training set consists of a total of 700 images, where 500 images are randomly chosen from the first 800 images in the UCID dataset [45] and 200 images are randomly chosen from the BSD-500’s training set [46].
    • Conclusion:

      The authors proposed a conditional GAN-based algorithm for the removal of rain streaks form a single image.
    • In comparison to the existing approaches which attempt to solve the de-raining problem in an image decomposition framework by using prior information, the authors investigated the use of conditional GANs for synthesizing de-rained image from a given input rainy image.
    • The proposed ID-CGAN method is compared against baseline configurations to illustrate the performance gains obtained by introducing the refined perceptual loss into the conditional GAN framework.
    • The authors aim to build upon the conditional GAN framework to overcome these drawback and investigate the possibility of using similar structures for solving related problems
    Tables
    • Table1: Compared to the existing methods, our ID-CGAN has several desirable properties: 1. No additional image processing. 2. Include discriminative factor into optimization. 3. Consider visual performance into optimization
    • Table2: Quantitative experiments evaluated on four different criterions
    • Table3: Quantitative results compared with three baseline configurations
    Download tables as Excel
    Funding
    • This work was supported by an ARO grant W911NF-16-1-
    Reference
    • X.-J. Mao, C. Shen, and Y.-B. Yang, “Image denoising using very deep fully convolutional encoder-decoder networks with symmetric skip connections,” arXiv preprint arXiv:1603.09056, 2016.
      Findings
    • H. Zhang and V. M. Patel, “Convolutional sparse coding-based image decomposition,” in British Machine Vision Conference, 2016.
      Google ScholarLocate open access versionFindings
    • J.-L. Starck, M. Elad, and D. L. Donoho, “Image decomposition via the combination of sparse representations and a variational approach,” Image Processing, IEEE Transactions on, vol. 14, no. 10, pp. 1570– 1582, 2005.
      Google ScholarLocate open access versionFindings
    • G. Peyre, J. Fadili, and J.-L. Starck, “Learning the morphological diversity,” SIAM Journal on Imaging Sciences, vol. 3, no. 3, pp. 646– 669, 2010.
      Google ScholarLocate open access versionFindings
    • K. Garg and S. K. Nayar, “Detection and removal of rain from videos,” in Computer Vision and Pattern Recognition, 2004. CVPR 2004. Proceedings of the 2004 IEEE Computer Society Conference on, vol.
      Google ScholarLocate open access versionFindings
    • ——, “Vision and rain,” International Journal of Computer Vision, vol. 75, no. 1, pp. 3–27, 2007.
      Google ScholarLocate open access versionFindings
    • X. Zhang, H. Li, Y. Qi, W. K. Leow, and T. K. Ng, “Rain removal in video by combining temporal and chromatic properties,” in 2006 IEEE International Conference on Multimedia and Expo. IEEE, 2006, pp. 461–464.
      Google ScholarLocate open access versionFindings
    • L.-W. Kang, C.-W. Lin, and Y.-H. Fu, “Automatic single-image-based rain streaks removal via image decomposition,” IEEE Transactions on Image Processing, vol. 21, no. 4, pp. 1742–1755, 2012.
      Google ScholarLocate open access versionFindings
    • Y. Luo, Y. Xu, and H. Ji, “Removing rain from a single image via discriminative sparse coding,” in Proceedings of the IEEE International Conference on Computer Vision, 2015, pp. 3397–3405.
      Google ScholarLocate open access versionFindings
    • Y. Li, R. T. Tan, X. Guo, J. Lu, and M. S. Brown, “Rain streak removal using layer priors,” in IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 2736–2744.
      Google ScholarLocate open access versionFindings
    • Y.-L. Chen and C.-T. Hsu, “A generalized low-rank appearance model for spatio-temporally correlated rain streaks,” in IEEE International Conference on Computer Vision, 2013, pp. 1968–1975.
      Google ScholarLocate open access versionFindings
    • X. Fu, J. Huang, X. Ding, Y. Liao, and J. Paisley, “Clearing the Skies: A deep network architecture for single-image rain removal,” ArXiv eprints, Sep. 2016.
      Google ScholarLocate open access versionFindings
    • D. Eigen, D. Krishnan, and R. Fergus, “Restoring an image taken through a window covered with dirt or rain,” in Proceedings of the IEEE International Conference on Computer Vision, 2013, pp. 633–640.
      Google ScholarLocate open access versionFindings
    • A. Radford, L. Metz, and S. Chintala, “Unsupervised representation learning with deep convolutional generative adversarial networks,” arXiv preprint arXiv:1511.06434, 2015.
      Findings
    • P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros, “Image-to-image translation with conditional adversarial networks,” arxiv, 2016.
      Google ScholarFindings
    • D. Pathak, P. Krahenbuhl, J. Donahue, T. Darrell, and A. A. Efros, “Context encoders: Feature learning by inpainting,” arXiv preprint arXiv:1604.07379, 2016.
      Findings
    • C. Ledig, L. Theis, F. Huszar, J. Caballero, A. Cunningham, A. Acosta, A. Aitken, A. Tejani, J. Totz, Z. Wang et al., “Photo-realistic single image super-resolution using a generative adversarial network,” arXiv preprint arXiv:1609.04802, 2016.
      Findings
    • H. Zhang and V. M. Patel, “Convolutional sparse and low-rank codingbased rain streak removal,” in 2017 IEEE Winter Conference on Applications of Computer Vision (WACV). IEEE, 2017, pp. 1–9.
      Google ScholarLocate open access versionFindings
    • W. Yang, R. T. Tan, J. Feng, J. Liu, Z. Guo, and S. Yan, “Joint rain detection and removal via iterative region dependent multi-task learning,” CoRR, vol. abs/1609.07769, 2016. [Online]. Available: http://arxiv.org/abs/1609.07769
      Findings
    • I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” in Advances in Neural Information Processing Systems, 2014, pp. 2672– 2680.
      Google ScholarLocate open access versionFindings
    • E. Thibodeau-Laufer, G. Alain, and J. Yosinski, “Deep generative stochastic networks trainable by backprop,” 2014.
      Google ScholarFindings
    • M. Mirza and S. Osindero, “Conditional generative adversarial nets,” arXiv preprint arXiv:1411.1784, 2014.
      Findings
    • L. Karacan, Z. Akata, A. Erdem, and E. Erdem, “Learning to generate images of outdoor scenes from attributes and semantic layouts,” arXiv preprint arXiv:1612.00215, 2016.
      Findings
    • T. Salimans, I. Goodfellow, W. Zaremba, V. Cheung, A. Radford, and X. Chen, “Improved techniques for training gans,” in Advances in Neural Information Processing Systems, 2016, pp. 2226–2234.
      Google ScholarLocate open access versionFindings
    • A. Creswell and A. A. Bharath, “Task specific adversarial cost function,” arXiv preprint arXiv:1609.08661, 2016.
      Findings
    • J. Zhao, M. Mathieu, and Y. LeCun, “Energy-based generative adversarial network,” arXiv preprint arXiv:1609.03126, 2016.
      Findings
    • C. Li and M. Wand, “Precomputed real-time texture synthesis with markovian generative adversarial networks,” in European Conference on Computer Vision, 2016, pp. 702–716.
      Google ScholarLocate open access versionFindings
    • D. Pathak, P. Krahenbuhl, J. Donahue, T. Darrell, and A. A. Efros, “Context encoders: Feature learning by inpainting,” in CVPR, 2016.
      Google ScholarFindings
    • H. Zhang, T. Xu, H. Li, S. Zhang, X. Huang, X. Wang, and D. Metaxas, “Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks,” arXiv preprint arXiv:1612.03242, 2016.
      Findings
    • S. Reed, Z. Akata, X. Yan, L. Logeswaran, B. Schiele, and H. Lee, “Generative adversarial text to image synthesis,” arXiv preprint arXiv:1605.05396, 2016.
      Findings
    • N. Jetchev, U. Bergmann, and R. Vollgraf, “Texture synthesis with spatial generative adversarial networks,” arXiv preprint arXiv:1611.08207, 2016.
      Findings
    • A. Dosovitskiy and T. Brox, “Generating images with perceptual similarity metrics based on deep networks,” arXiv preprint arXiv:1602.02644, 2016.
      Findings
    • J. Long, E. Shelhamer, and T. Darrell, “Fully convolutional networks for semantic segmentation,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015, pp. 3431–3440.
      Google ScholarLocate open access versionFindings
    • D. Eigen and R. Fergus, “Predicting depth, surface normals and semantic labels with a common multi-scale convolutional architecture,” in Proceedings of the IEEE International Conference on Computer Vision, 2015, pp. 2650–2658.
      Google ScholarLocate open access versionFindings
    • A. Mahendran and A. Vedaldi, “Understanding deep image representations by inverting them,” in 2015 IEEE conference on computer vision and pattern recognition (CVPR). IEEE, 2015, pp. 5188–5196.
      Google ScholarLocate open access versionFindings
    • S. Zheng, S. Jayasumana, B. Romera-Paredes, V. Vineet, Z. Su, D. Du, C. Huang, and P. H. Torr, “Conditional random fields as recurrent neural networks,” in Proceedings of the IEEE International Conference on Computer Vision, 2015, pp. 1529–1537.
      Google ScholarLocate open access versionFindings
    • A. Dosovitskiy and T. Brox, “Inverting visual representations with convolutional networks,” arXiv preprint arXiv:1506.02753, 2015.
      Findings
    • L. A. Gatys, A. S. Ecker, and M. Bethge, “A neural algorithm of artistic style,” arXiv preprint arXiv:1508.06576, 2015.
      Findings
    • J. Johnson, A. Alahi, and L. Fei-Fei, “Perceptual losses for real-time style transfer and super-resolution,” arXiv preprint arXiv:1603.08155, 2016.
      Findings
    • J. Bobin, J. L. Starck, J. M. Fadili, Y. Moudden, and D. L. Donoho, “Morphological component analysis: An adaptive thresholding strategy,” IEEE Transactions on Image Processing, vol. 16, no. 11, pp. 2675–2681, Nov 2007.
      Google ScholarLocate open access versionFindings
    • J. Xie, L. Xu, and E. Chen, “Image denoising and inpainting with deep neural networks,” in Advances in Neural Information Processing Systems, 2012, pp. 341–349.
      Google ScholarLocate open access versionFindings
    • C. Dong, C. C. Loy, and X. Tang, “Accelerating the super-resolution convolutional neural network,” in European Conference on Computer Vision. Springer, 2016, pp. 391–407.
      Google ScholarFindings
    • J. Johnson, A. Alahi, and L. Fei-Fei, “Perceptual losses for real-time style transfer and super-resolution,” arXiv preprint arXiv:1603.08155, 2016.
      Findings
    • K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” arXiv preprint arXiv:1409.1556, 2014.
      Findings
    • G. Schaefer and M. Stich, “Ucid: an uncompressed color image database,” in Electronic Imaging 2004. International Society for Optics and Photonics, 2003, pp. 472–480.
      Google ScholarLocate open access versionFindings
    • P. Arbelaez, M. Maire, C. Fowlkes, and J. Malik, “Contour detection and hierarchical image segmentation,” IEEE transactions on pattern analysis and machine intelligence, vol. 33, no. 5, pp. 898–916, 2011.
      Google ScholarLocate open access versionFindings
    • Y. Zhang, Z. Jiang, and L. Davis, “Learning structured low-rank representations for image classification,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2013, pp. 676– 683.
      Google ScholarLocate open access versionFindings
    • Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: from error visibility to structural similarity,” IEEE transactions on image processing, vol. 13, no. 4, pp. 600–612, 2004.
      Google ScholarLocate open access versionFindings
    • Z. Wang and A. C. Bovik, “A universal image quality index,” IEEE signal processing letters, vol. 9, no. 3, pp. 81–84, 2002.
      Google ScholarLocate open access versionFindings
    • H. R. Sheikh and A. C. Bovik, “Image information and visual quality,” IEEE Transactions on Image Processing, vol. 15, no. 2, pp. 430–444, 2006.
      Google ScholarLocate open access versionFindings
    • R. Collobert, K. Kavukcuoglu, and C. Farabet, “Torch7: A matlab-like environment for machine learning,” in BigLearn, NIPS Workshop, 2011.
      Google ScholarLocate open access versionFindings
    • D. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980, 2014.
      Findings
    Your rating :
    0

     

    Tags
    Comments