CycleISP: Real Image Restoration via Improved Data Synthesis

CVPR, pp. 2693-2702, 2020.

Cited by: 3|Bibtex|Views25|Links
EI
Keywords:
image restorationchannel attentionMulti-Layer Perceptronimage signal processingneural networkMore(21+)
Weibo:
We demonstrate that the CycleISP model can be applied to the color matching problem in stereoscopic cinema

Abstract:

The availability of large-scale datasets has helped unleash the true potential of deep convolutional neural networks (CNNs). However, for the single-image denoising problem, capturing a real dataset is an unacceptably expensive and cumbersome procedure. Consequently, image denoising algorithms are mostly developed and evaluated on synth...More

Code:

Data:

0
Introduction
  • High-level computer vision tasks, such as image classification, object detection and segmentation have witnessed significant progress due to deep CNNs [33].
  • Spatial pixels misalignment, color and brightness mismatch is inevitable due to changes in lighting conditions and camera/object motion.
  • This expensive and cumbersome exercise of acquiring image pairs needs to be repeated with different camera sensors, as they exhibit different noise characteristics
Highlights
  • High-level computer vision tasks, such as image classification, object detection and segmentation have witnessed significant progress due to deep convolutional neural networks [33]
  • In this paper we propose a CycleISP framework that converts sRGB images to RAW data, and back to sRGB images, without requiring any knowledge of camera parameters
  • We present a framework that is capable of synthesizing realistic noise data for training convolutional neural networks to effectively remove noise from RAW as well as sRGB images
  • The numbers in the sRGB super column are provided by the server after passing the denoised RAW images through the camera imaging pipeline [32] using image metadata
  • Conclusion sIn this work, we propose a data-driven CycleISP framework that is capable of converting sRGB images to RAW data and back to sRGB images
  • We demonstrate that the CycleISP model can be applied to the color matching problem in stereoscopic cinema
Methods
  • TNRD* [14] MLP* [9] FoE [49] EPLL* [67] KSVD* [3] WNNM* [27] NCSR* [18] BM3D* [15] DnCNN [64] N3Net [45] UPI (Raw) [7] Ours

    RAW PSNR ↑ SSIM ↑

    sRGB PSNR ↑ SSIM ↑

    EPLL [67] GLIDE [56] TNRD [14] FoE [49] MLP [9] KSVD [3] DnCNN [64] NLM [8] WNNM [27] BM3D [15]

    0.918 dataset [31] and split them into a ratio of 90:5:5 for training, validation and testing.
  • TNRD* [14] MLP* [9] FoE [49] EPLL* [67] KSVD* [3] WNNM* [27] NCSR* [18] BM3D* [15] DnCNN [64] N3Net [45] UPI (Raw) [7] Ours.
  • The authors synthesize clean/noisy paired training data using the procedure described in Section 4
Results
  • The authors evaluate the denoising results of the proposed CycleISP model with existing state-of-theart methods on the RAW data from DND [44] and SIDD [1] benchmarks.
  • Table 1 shows the quantitative results (PSNR/SSIM) of all competing methods on the DND dataset obtained from the website of the evaluation server [16].
  • The numbers in the sRGB super column are provided by the server after passing the denoised RAW images through the camera imaging pipeline [32] using image metadata.
  • The authors' algorithm achieves 6.89 dB improvement in PSNR over the BM3D algorithm[15]
Conclusion
  • SIn this work, the authors propose a data-driven CycleISP framework that is capable of converting sRGB images to RAW data and back to sRGB images.
  • The CycleISP model allows them to synthesize realistic clean/noisy paired training data both in RAW and sRGB spaces.
  • The authors demonstrate that the CycleISP model can be applied to the color matching problem in stereoscopic cinema.
  • The authors' future work includes exploring and extending the CycleISP model for other low-level vision problems such as super-resolution and deblurring.
Summary
  • Introduction:

    High-level computer vision tasks, such as image classification, object detection and segmentation have witnessed significant progress due to deep CNNs [33].
  • Spatial pixels misalignment, color and brightness mismatch is inevitable due to changes in lighting conditions and camera/object motion.
  • This expensive and cumbersome exercise of acquiring image pairs needs to be repeated with different camera sensors, as they exhibit different noise characteristics
  • Methods:

    TNRD* [14] MLP* [9] FoE [49] EPLL* [67] KSVD* [3] WNNM* [27] NCSR* [18] BM3D* [15] DnCNN [64] N3Net [45] UPI (Raw) [7] Ours

    RAW PSNR ↑ SSIM ↑

    sRGB PSNR ↑ SSIM ↑

    EPLL [67] GLIDE [56] TNRD [14] FoE [49] MLP [9] KSVD [3] DnCNN [64] NLM [8] WNNM [27] BM3D [15]

    0.918 dataset [31] and split them into a ratio of 90:5:5 for training, validation and testing.
  • TNRD* [14] MLP* [9] FoE [49] EPLL* [67] KSVD* [3] WNNM* [27] NCSR* [18] BM3D* [15] DnCNN [64] N3Net [45] UPI (Raw) [7] Ours.
  • The authors synthesize clean/noisy paired training data using the procedure described in Section 4
  • Results:

    The authors evaluate the denoising results of the proposed CycleISP model with existing state-of-theart methods on the RAW data from DND [44] and SIDD [1] benchmarks.
  • Table 1 shows the quantitative results (PSNR/SSIM) of all competing methods on the DND dataset obtained from the website of the evaluation server [16].
  • The numbers in the sRGB super column are provided by the server after passing the denoised RAW images through the camera imaging pipeline [32] using image metadata.
  • The authors' algorithm achieves 6.89 dB improvement in PSNR over the BM3D algorithm[15]
  • Conclusion:

    SIn this work, the authors propose a data-driven CycleISP framework that is capable of converting sRGB images to RAW data and back to sRGB images.
  • The CycleISP model allows them to synthesize realistic clean/noisy paired training data both in RAW and sRGB spaces.
  • The authors demonstrate that the CycleISP model can be applied to the color matching problem in stereoscopic cinema.
  • The authors' future work includes exploring and extending the CycleISP model for other low-level vision problems such as super-resolution and deblurring.
Tables
  • Table1: RAW denoising results on the DND benchmark dataset [<a class="ref-link" id="c44" href="#r44">44</a>]. * denotes that these methods use variance stabilizing transform (VST) [<a class="ref-link" id="c40" href="#r40">40</a>] to provide their best results
  • Table2: RAW denoising results on the SIDD dataset [<a class="ref-link" id="c1" href="#r1">1</a>]
  • Table3: Denoising sRGB images of the DND benchmark dataset [<a class="ref-link" id="c44" href="#r44">44</a>]
  • Table4: Denoising sRGB images of the SIDD benchmark dataset [<a class="ref-link" id="c1" href="#r1">1</a>]
  • Table5: Generalization Test. U-Net model is trained only for DND [<a class="ref-link" id="c44" href="#r44">44</a>] with our technique and with the UPI [<a class="ref-link" id="c7" href="#r7">7</a>] method, and directly evaluated on the SIDD dataset [<a class="ref-link" id="c1" href="#r1">1</a>]
  • Table6: Ablation study: RAW2RGB branch
  • Table7: Layout of SA and CA in DAB
Download tables as Excel
Related work
  • The presence of noise in images is inevitable, irrespective of the acquisition method; now more than ever, when majority of images come from smartphone cameras having small sensor size but large resolution. Single-image denoising is a vastly researched problem in the computer vision and image processing community, with early works dating back to 1960’s [6]. Classic methods on denoising are mainly based on the following two principles. (1) Modifying transform coefficients using the DCT [61], wavelets [19, 54], etc. (2) Averaging neighborhood values: in all directions using Gaussian kernel, in all directions only if pixels have similar values [55, 57] and along contours [42, 50].

    While these aforementioned methods provide satisfactory results in terms of image fidelity metrics and visual quality, the Non-local Means (NLM) algorithm of Buades et al [8] makes significant advances in denoising. The NLM method exploits the redundancy, or self-similarity [20] present in natural images. For many years the patch-based methods yielded comparable results, thus prompting studies [11, 12, 37] to investigate whether we reached the theoretical limits of denoising performance. Subsequently, Burger et al [9] train a simple Multi-Layer Perceptron (MLP) on a large synthetic noise dataset. This method performs well against previous sophisticated algorithms. Several recent methods use deep CNNs [4, 7, 25, 28, 45, 64, 65, 2] and demonstrate promising denoising performance.
Funding
  • Ming-Hsuan Yang is supported by the NSF CAREER Grant 149783
Reference
  • Abdelrahman Abdelhamed, Stephen Lin, and Michael S Brown. A high-quality denoising dataset for smartphone cameras. In CVPR, 2018. 2, 5, 6, 7, 8
    Google ScholarLocate open access versionFindings
  • Abdelrahman Abdelhamed, Radu Timofte, and Michael S Brown. Ntire 2019 challenge on real image denoising: Methods and results. In CVPRW, 2019. 2
    Google ScholarLocate open access versionFindings
  • Michal Aharon, Michael Elad, and Alfred Bruckstein. KSVD: an algorithm for designing overcomplete dictionaries for sparse representation. Trans. Sig. Proc., 2006. 6, 7
    Google ScholarLocate open access versionFindings
  • Saeed Anwar and Nick Barnes. Real image denoising with feature attention. ICCV, 2019. 2, 7
    Google ScholarLocate open access versionFindings
  • Marcelo Bertalmıo. Image Processing for Cinema. CRC Press, 2014. 8
    Google ScholarFindings
  • Marcelo Bertalmıo. Denoising of Photographic Images and Video. Springer, 2018. 2
    Google ScholarFindings
  • Tim Brooks, Ben Mildenhall, Tianfan Xue, Jiawen Chen, Dillon Sharlet, and Jonathan T Barron. Unprocessing images for learned raw denoising. In CVPR, 2019. 1, 2, 3, 4, 5, 6, 7, 8
    Google ScholarLocate open access versionFindings
  • Antoni Buades, Bartomeu Coll, and J-M Morel. A non-local algorithm for image denoising. In CVPR, 2005. 2, 6, 7
    Google ScholarLocate open access versionFindings
  • Harold C Burger, Christian J Schuler, and Stefan Harmeling. Image denoising: Can plain neural networks compete with BM3D? In CVPR, 2012. 2, 5, 6, 7
    Google ScholarLocate open access versionFindings
  • Vladimir Bychkovsky, Sylvain Paris, Eric Chan, and Fredo Durand. Learning photographic global tonal adjustment with a database of input/output image pairs. In CVPR, 2011. 4, 6
    Google ScholarLocate open access versionFindings
  • Priyam Chatterjee and Peyman Milanfar. Is denoising dead? TIP, 2009. 2
    Google ScholarFindings
  • Priyam Chatterjee and Peyman Milanfar. Fundamental limits of image denoising: are we there yet? In ICASSP, 2010. 2
    Google ScholarLocate open access versionFindings
  • Chen Chen, Qifeng Chen, Jia Xu, and Vladlen Koltun. Learning to see in the dark. In CVPR, 2018. 4
    Google ScholarLocate open access versionFindings
  • Yunjin Chen, Wei Yu, and Thomas Pock. On learning optimized reaction diffusion processes for effective image restoration. In CVPR, 2015. 6, 7
    Google ScholarLocate open access versionFindings
  • Kostadin Dabov, Alessandro Foi, Vladimir Katkovnik, and Karen Egiazarian. Image denoising by sparse 3-D transformdomain collaborative filtering. TIP, 2007. 2, 6, 7
    Google ScholarLocate open access versionFindings
  • https://noise.visinf.tu-darmstadt.de/benchmark/, 2017.[Online; accessed 15-Nov-2019].6
    Findings
  • J. Deng, W. Dong, R. Socher, L. Li, Kai Li, and Li FeiFei. ImageNet: A large-scale hierarchical image database. In CVPR, 2009. 1
    Google ScholarLocate open access versionFindings
  • Weisheng Dong, Lei Zhang, Guangming Shi, and Xin Li. Nonlocally centralized sparse representation for image restoration. TIP, 2012. 6, 7
    Google ScholarLocate open access versionFindings
  • David L Donoho. De-noising by soft-thresholding. Trans. on information theory, 1995. 2
    Google ScholarLocate open access versionFindings
  • Alexei A Efros and Thomas K Leung. Texture synthesis by non-parametric sampling. In ICCV, 1999. 2
    Google ScholarLocate open access versionFindings
  • Gabriel Eilertsen, Joel Kronander, Gyorgy Denes, Rafał K Mantiuk, and Jonas Unger. HDR image reconstruction from a single exposure using deep cnns. TOG, 2017. 4
    Google ScholarLocate open access versionFindings
  • Alessandro Foi. Clipped noisy images: Heteroskedastic modeling and practical denoising. Signal Processing, 2009. 2
    Google ScholarLocate open access versionFindings
  • Alessandro Foi, Sakari Alenius, Vladimir Katkovnik, and Karen Egiazarian. Noise measurement for raw-data of digital imaging sensors by automatic segmentation of nonuniform targets. Sensors, 2007. 2
    Google ScholarLocate open access versionFindings
  • Alessandro Foi, Mejdi Trimeche, Vladimir Katkovnik, and Karen Egiazarian. Practical poissonian-gaussian noise modeling and fitting for single-image raw-data. TIP, 2008. 2
    Google ScholarFindings
  • Michael Gharbi, Gaurav Chaurasia, Sylvain Paris, and Fredo Durand. Deep joint demosaicking and denoising. TOG, 2016. 2, 4
    Google ScholarLocate open access versionFindings
  • Gabriela Ghimpeteanu, Thomas Batard, Tamara Seybold, and Marcelo Bertalmıo. Local denoising applied to raw images may outperform non-local patch-based methods applied to the camera output. In Electronic Imaging, 2016. 2, 7
    Google ScholarLocate open access versionFindings
  • Shuhang Gu, Lei Zhang, Wangmeng Zuo, and Xiangchu Feng. Weighted nuclear norm minimization with application to image denoising. In CVPR, 2014. 6, 7
    Google ScholarLocate open access versionFindings
  • Shi Guo, Zifei Yan, Kai Zhang, Wangmeng Zuo, and Lei Zhang. Toward convolutional blind denoising of real photographs. In CVPR, 2019. 2, 7
    Google ScholarLocate open access versionFindings
  • Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016. 4
    Google ScholarLocate open access versionFindings
  • Jie Hu, Li Shen, and Gang Sun. Squeeze-and-excitation networks. In CVPR, 2018. 4, 5
    Google ScholarLocate open access versionFindings
  • Mark J Huiskes, Bart Thomee, and Michael S Lew. New trends and ideas in visual concept detection: the MIR flickr retrieval evaluation initiative. In ACM MIR, 2010. 6
    Google ScholarLocate open access versionFindings
  • Hakki Can Karaimer and Michael S Brown. A software platform for manipulating the camera imaging pipeline. In ECCV, 2016. 6
    Google ScholarLocate open access versionFindings
  • Salman Khan, Hossein Rahmani, Syed Afaq Ali Shah, and Mohammed Bennamoun. A guide to convolutional neural networks for computer vision. Synthesis Lectures on Computer Vision, 8(1):1–207, 2018. 1
    Google ScholarLocate open access versionFindings
  • Hiroaki Kotera. A scene-referred color transfer for pleasant imaging on display. In ICIP, 2005. 8
    Google ScholarLocate open access versionFindings
  • Marc Lebrun, Miguel Colom, Antoni Buades, and JeanMichel Morel. Secrets of image denoising cuisine. Acta Numerica, 2012. 2
    Google ScholarLocate open access versionFindings
  • Marc Lebrun, Miguel Colom, and Jean-Michel Morel. The noise clinic: a blind image denoising algorithm. Image Processing On Line, 2015. 7
    Google ScholarLocate open access versionFindings
  • Anat Levin and Boaz Nadler. Natural image denoising: Optimality and inherent bounds. In CVPR, 2011. 2
    Google ScholarLocate open access versionFindings
  • Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollar, and C Lawrence Zitnick. Microsoft COCO: Common objects in context. In ECCV, 2014. 1
    Google ScholarLocate open access versionFindings
  • Jiaming Liu, Chi-Hao Wu, Yuzhi Wang, Qin Xu, Yuqian Zhou, et al. Learning raw image denoising with bayer pattern unification and bayer preserving augmentation. In CVPR Workshops, 2019. 4, 6
    Google ScholarLocate open access versionFindings
  • Markku Makitalo and Alessandro Foi. Optimal inversion of the generalized anscombe transformation for poissongaussian noise. TIP, 2012. 6
    Google ScholarLocate open access versionFindings
  • Bernard Mendiburu. 3D Movie Making: Stereoscopic Digital Cinema from Script to Screen. Focal Press, 2009. 8
    Google ScholarFindings
  • Pietro Perona and Jitendra Malik. Scale-space and edge detection using anisotropic diffusion. TPAMI, 1990. 2
    Google ScholarLocate open access versionFindings
  • Francois Pitie, Anil C Kokaram, and Rozenn Dahyot. Automated colour grading using colour distribution transfer. Trans. on CVIU, 2007. 8
    Google ScholarLocate open access versionFindings
  • Tobias Plotz and Stefan Roth. Benchmarking denoising algorithms with real photographs. In CVPR, 2017. 1, 2, 5, 6, 7, 8
    Google ScholarLocate open access versionFindings
  • Tobias Plotz and Stefan Roth. Neural nearest neighbors networks. In NeurIPS, 2018. 1, 2, 6
    Google ScholarLocate open access versionFindings
  • R. Ramanath, W. E. Snyder, Y. Yoo, and M. S. Drew. Color image processing pipeline. IEEE Signal Processing Magazine, 2005. 2, 3
    Google ScholarLocate open access versionFindings
  • Erik Reinhard, Michael Adhikhmin, Bruce Gooch, and Peter Shirley. Color transfer between images. Trans. on Computer graphics and applications, 2001. 8
    Google ScholarLocate open access versionFindings
  • Dongwei Ren, Wangmeng Zuo, Qinghua Hu, Pengfei Zhu, and Deyu Meng. Progressive image deraining networks: a better and simpler baseline. In CVPR, 2019. 4
    Google ScholarLocate open access versionFindings
  • Stefan Roth and Michael J Black. Fields of experts. IJCV, 2009. 6, 7
    Google ScholarLocate open access versionFindings
  • Leonid I Rudin, Stanley Osher, and Emad Fatemi. Nonlinear total variation based noise removal algorithms. Physica D: nonlinear phenomena, 1992. 2
    Google ScholarLocate open access versionFindings
  • Eli Schwartz, Raja Giryes, and Alex M Bronstein. DeepISP: Towards learning an end-to-end image processing pipeline. TIP, 2018. 4
    Google ScholarLocate open access versionFindings
  • Tamara Seybold, Ozlem Cakmak, Christian Keimel, and Walter Stechele. Noise characteristics of a single sensor camera in digital color image processing. In CIC, 2014. 2
    Google ScholarLocate open access versionFindings
  • Wenzhe Shi, Jose Caballero, Ferenc Huszar, Johannes Totz, Andrew P Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In CVPR, 2016. 4
    Google ScholarLocate open access versionFindings
  • Eero P Simoncelli and Edward H Adelson. Noise removal via bayesian wavelet coring. In ICIP, 1996. 2
    Google ScholarLocate open access versionFindings
  • Stephen M Smith and J Michael Brady. SUSANa new approach to low level image processing. IJCV, 1997. 2
    Google ScholarLocate open access versionFindings
  • Hossein Talebi and Peyman Milanfar. Global image denoising. TIP, 2013. 6, 7
    Google ScholarLocate open access versionFindings
  • Carlo Tomasi and Roberto Manduchi. Bilateral filtering for gray and color images. In ICCV, 1998. 2
    Google ScholarLocate open access versionFindings
  • Sanghyun Woo, Jongchan Park, Joon-Young Lee, and In So Kweon. CBAM: Convolutional block attention module. In ECCV, 2018. 4
    Google ScholarLocate open access versionFindings
  • Jun Xu, Lei Zhang, and David Zhang. A trilateral weighted sparse coding scheme for real-world image denoising. In ECCV, 2018. 7
    Google ScholarLocate open access versionFindings
  • Jun Xu, Lei Zhang, David Zhang, and Xiangchu Feng. Multi-channel weighted nuclear norm minimization for real color image denoising. In ICCV, 2017. 7
    Google ScholarLocate open access versionFindings
  • Leonid P Yaroslavsky. Local adaptive image restoration and enhancement with the use of DFT and DCT in a running window. In Wavelet Applications in Signal and Image Processing IV, 1996. 2
    Google ScholarLocate open access versionFindings
  • Syed Waqas Zamir, Aditya Arora, Salman Khan, Fahad Shahbaz Khan, and Ling Shao. Learning digital camera pipeline for extreme low-light imaging. arXiv preprint arXiv:1904.05939, 2019. 4
    Findings
  • He Zhang, Vishwanath Sindagi, and Vishal M Patel. Multiscale single image dehazing using perceptual pyramid deep network. In CVPR Workshops, 2018. 4
    Google ScholarLocate open access versionFindings
  • Kai Zhang, Wangmeng Zuo, Yunjin Chen, Deyu Meng, and Lei Zhang. Beyond a gaussian denoiser: Residual learning of deep cnn for image denoising. TIP, 2017. 2, 4, 5, 6, 7
    Google ScholarLocate open access versionFindings
  • Kai Zhang, Wangmeng Zuo, and Lei Zhang. FFDNet: Toward a fast and flexible solution for CNN-based image denoising. TIP, 2018. 2, 7
    Google ScholarLocate open access versionFindings
  • Yulun Zhang, Kunpeng Li, Kai Li, Lichen Wang, Bineng Zhong, and Yun Fu. Image super-resolution using very deep residual channel attention networks. In ECCV, 2018. 4
    Google ScholarLocate open access versionFindings
  • Daniel Zoran and Yair Weiss. From learning models of natural image patches to whole image restoration. In ICCV, 2011. 6, 7
    Google ScholarLocate open access versionFindings
Your rating :
0

 

Tags
Comments