Unsupervised Adaptation Learning for Hyperspectral Imagery Super-Resolution

    CVPR 2020, 2020.

    Cited by: 0|Bibtex|Views27|Links
    Keywords:
    multispectral imageunknown degenerationroot-mean-square errorspectral angle mappermaximum a posterioriMore(10+)
    Wei bo:
    We present an unsupervised adaptation learning framework for fusionbased hyperspectral image SR, which shows good generalization performance on different datasets and unknown degenerations

    Abstract:

    The key for fusion based hyperspectral image (HSI) super-resolution (SR) is to infer the posteriori of a latent HSI using appropriate image prior and likelihood that depends on degeneration. However, in practice the priors of high-dimensional HSIs can be extremely complicated and the degeneration is often unknown. Consequently most existi...More

    Code:

    Data:

    0
    Introduction
    • Hyperspectral images (HSIs) consists of hundreds of spectral bands that record the reflectance of an imaging scene across a consecutive wavelengths with narrow interval (e.g. 10nm) [1, 32], where each pixel contains a spec-

      trum.
    • The key lies on inferring the posteriori of the latent HSI using an appropriate image prior and the likelihood determined by the degeneration from the latent HSI to the observed LR HSI.
    • To this end, existing approaches have handcrafted various shallow prior models, e.g. sparsity [12, 8], low-rank [30] etc.
    • When applied to real scenarios, most existing approaches fail to generalize appropriately or obtain pleasing SR performance, as shown in Figure 1
    Highlights
    • Hyperspectral images (HSIs) consists of hundreds of spectral bands that record the reflectance of an imaging scene across a consecutive wavelengths with narrow interval (e.g. 10nm) [1, 32], where each pixel contains a spec-

      trum
    • The key lies on inferring the posteriori of the latent hyperspectral image using an appropriate image prior and the likelihood determined by the degeneration from the latent hyperspectral image to the observed low resolution hyperspectral image
    • We propose an unsupervised adaptation learning (UAL) framework and demonstrate good generalization efficacy in hyperspectral image SR
    • Previous work [33] has shown that the degenerated observation often contains some image-specific statistics, and can supervise the learning of an image-specific prior when the degeneration is given [21]. To generalize this idea to real cases with unknown degenerations, we introduce a degeneration network together with the given spectral response function P to map the image Z generated from the fusion module back into the observed X and Y, respectively, as shown in Figure 2(a)
    • The concatenation enables the correlation between two observed images to be exploited. It fails to explicitly leverage specific knowledge from each input. To sufficiently utilize both intra-input and inter-input knowledge, we propose a mutual-guiding fusion module that takes three different inputs, including the high resolution multispectral image X, low resolution hyperspectral image Y 1 and their concatenation [X; Y]
    • We present an unsupervised adaptation learning framework for fusionbased hyperspectral image SR, which shows good generalization performance on different datasets and unknown degenerations
    Methods
    • The authors select five state-of-the-art fusion based HSI SR methods, including NSSR [8], CMS [30], MHF-net [25], YONG [15] and DIP [21].
    • RMSE PSNR SAM SSIM RMSE PSNR SAM SSIM RMSE PSNR SAM SSIM NSSR [8] CMS [30] DIP [21] YONG [15] UAL CAVE Havard NTIRE.
    • The authors find that the proposed UAL recovers more details and obtain the best image quality
    Conclusion
    • The authors present an UAL framework for fusionbased HSI SR, which shows good generalization performance on different datasets and unknown degenerations.
    • The model benefits from implicitly learning a deep imagespecific prior as well as estimating the unknown degeneration.
    • To this end, the authors first develop a two-stage SR network and pretrain the first stage in a supervised manner.
    • It provides a promising way to generalize deep models to unseen cases in practice, which can be applied to other image restoration tasks, such as image denoising, deblurring etc
    Summary
    • Introduction:

      Hyperspectral images (HSIs) consists of hundreds of spectral bands that record the reflectance of an imaging scene across a consecutive wavelengths with narrow interval (e.g. 10nm) [1, 32], where each pixel contains a spec-

      trum.
    • The key lies on inferring the posteriori of the latent HSI using an appropriate image prior and the likelihood determined by the degeneration from the latent HSI to the observed LR HSI.
    • To this end, existing approaches have handcrafted various shallow prior models, e.g. sparsity [12, 8], low-rank [30] etc.
    • When applied to real scenarios, most existing approaches fail to generalize appropriately or obtain pleasing SR performance, as shown in Figure 1
    • Methods:

      The authors select five state-of-the-art fusion based HSI SR methods, including NSSR [8], CMS [30], MHF-net [25], YONG [15] and DIP [21].
    • RMSE PSNR SAM SSIM RMSE PSNR SAM SSIM RMSE PSNR SAM SSIM NSSR [8] CMS [30] DIP [21] YONG [15] UAL CAVE Havard NTIRE.
    • The authors find that the proposed UAL recovers more details and obtain the best image quality
    • Conclusion:

      The authors present an UAL framework for fusionbased HSI SR, which shows good generalization performance on different datasets and unknown degenerations.
    • The model benefits from implicitly learning a deep imagespecific prior as well as estimating the unknown degeneration.
    • To this end, the authors first develop a two-stage SR network and pretrain the first stage in a supervised manner.
    • It provides a promising way to generalize deep models to unseen cases in practice, which can be applied to other image restoration tasks, such as image denoising, deblurring etc
    Tables
    • Table1: Effect of each component in the proposed UAL
    • Table2: Performance under degeneration produced by different kernels and 40db noise with SR scale s=8
    • Table3: Performance under degeneration produced by different levels of noise corruption and kernel k1 with SR scale s=8
    • Table4: Performance when SR scale s is 8 and the degeneration is produced by k1 and 40db noise
    • Table5: Performance when SR scale s is 32 and the degeneration is produced by k1 and 40db noise
    • Table6: No-reference HSI quality measurement score [<a class="ref-link" id="c26" href="#r26">26</a>] of each method on the real HSI SR task. (The smaller the better)
    • Table7: Average runtime on CAVE dataset when SR scale s is 32 and the degeneration is produced by k1 with 40db noise
    Download tables as Excel
    Related work
    • According to the image prior utilized, existing fusionbased SR methods can be divided into two categories.

      Shallow image prior Existing methods have handcrafted various shallow image prior models [24, 3, 12, 8, 30]. For example, Wycoff et al [24] and Lanaras [12] propose to decompose the latent HR HSI into a non-negative endmember matrix and an abundance matrix, and then impose a sparse prior on the abundance matrix. Akhtar [2] exploit the signal sparsity, nonnegativity and spatial structure of the latent HSI by projecting each spectrum onto a pre-extracted spectral dictionary. In [3], a Bayesian sparse representation scheme is utilized to infer the probability distributions of spectra and their proportions decomposed from the latent HSI. Recently, Dong et al [8] further consider the spatially non-local similarity of the latent HSI. Zhang et al [30]
    Funding
    • This work was supported in part by the National Natural Science Foundation of China(No 61671385), in part by the Science, Technology and Innovation Commission of Shenzhen Manicipality(No.JCYJ20190806160210899)
    Reference
    • Naveed Akhtar and Ajmal S Mian. Hyperspectral recovery from rgb images using gaussian processes. IEEE transactions on pattern analysis and machine intelligence, 2018.
      Google ScholarFindings
    • Naveed Akhtar, Faisal Shafait, and Ajmal Mian. Sparse spatio-spectral representation for hyperspectral image superresolution. In European Conference on Computer Vision, pages 63–78.
      Google ScholarLocate open access versionFindings
    • Naveed Akhtar, Faisal Shafait, and Ajmal Mian. Bayesian sparse representation for hyperspectral image super resolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3631–3640, 2015.
      Google ScholarLocate open access versionFindings
    • Naveed Akhtar, Faisal Shafait, and Ajmal Mian. Hierarchical beta process with gaussian process prior for hyperspectral image super resolution. In European Conference on Computer Vision, pages 103–120.
      Google ScholarLocate open access versionFindings
    • Rushil Anirudh, Jayaraman J Thiagarajan, Bhavya Kailkhura, and Timo Bremer. An unsupervised approach to solving inverse problems using generative adversarial networks. arXiv preprint arXiv:1805.07281, 2018.
      Findings
    • Ayan Chakrabarti and Todd Zickler. Statistics of real-world hyperspectral images. In CVPR 2011, pages 193–200. IEEE, 2011.
      Google ScholarLocate open access versionFindings
    • Renwei Dian, Leyuan Fang, and Shutao Li. Hyperspectral image super-resolution via non-local sparse tensor factorization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5344–5353, 2017.
      Google ScholarLocate open access versionFindings
    • Weisheng Dong, Fazuo Fu, Guangming Shi, Xun Cao, Jinjian Wu, Guangyu Li, and Xin Li. Hyperspectral image super-resolution via non-negative structured sparse representation. IEEE Transactions on Image Processing, 25(5):2337– 2352, 2016.
      Google ScholarLocate open access versionFindings
    • Netalee Efrat, Daniel Glasner, Alexander Apartsin, Boaz Nadler, and Anat Levin. Accurate blur models vs. image priors in single image super-resolution. In Proceedings of the IEEE International Conference on Computer Vision, pages 2832–2839, 2013.
      Google ScholarLocate open access versionFindings
    • Charilaos I Kanatsoulis, Xiao Fu, Nicholas D Sidiropoulos, and Wing-Kin Ma. Hyperspectral super-resolution via coupled tensor factorization: Identifiability and algorithms. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 3191–3195. IEEE, 2018.
      Google ScholarLocate open access versionFindings
    • Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
      Findings
    • Charis Lanaras, Emmanuel Baltsavias, and Konrad Schindler. Hyperspectral super-resolution by coupled spectral unmixing. In Proceedings of the IEEE International Conference on Computer Vision, pages 3586–3594, 2015.
      Google ScholarLocate open access versionFindings
    • Jiaojiao Li, Xi Zhao, Yunsong Li, Qian Du, Bobo Xi, and Jing Hu. Classification of hyperspectral imagery using a new fully convolutional neural network. IEEE Geoscience and Remote Sensing Letters, 15(2):292–296, 2018.
      Google ScholarLocate open access versionFindings
    • Shutao Li, Renwei Dian, Leyuan Fang, and Jose M BioucasDias. Fusing hyperspectral and multispectral images via coupled sparse tensor factorization. IEEE Transactions on Image Processing, 27(8):4118–4130, 2018.
      Google ScholarLocate open access versionFindings
    • Yong Li, Lei Zhang, Chunna Tian, Chen Ding, Yanning Zhang, and Wei Wei. Hyperspectral image super-resolution extending: An effective fusion based method without knowing the spatial transformation matrix. In 2017 IEEE International Conference on Multimedia and Expo (ICME), pages 1117–1122. IEEE, 2017.
      Google ScholarLocate open access versionFindings
    • Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th international conference on machine learning (ICML10), pages 807–814, 2010.
      Google ScholarLocate open access versionFindings
    • Nasser M Nasrabadi. Hyperspectral target detection: An overview of current and future challenges. IEEE Signal Processing Magazine, 31(1):34–44, 2013.
      Google ScholarLocate open access versionFindings
    • Yi Peng, Deyu Meng, Zongben Xu, Chenqiang Gao, Yi Yang, and Biao Zhang. Decomposable nonlocal tensor dictionary learning for multispectral image denoising. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2949–2956, 2014.
      Google ScholarLocate open access versionFindings
    • Ying Qu, Hairong Qi, and Chiman Kwan. Unsupervised sparse dirichlet-net for hyperspectral image super-resolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2511–2520, 2018.
      Google ScholarLocate open access versionFindings
    • Radu Timofte, Shuhang Gu, Jiqing Wu, and Luc Van Gool. Ntire 2018 challenge on single image super-resolution: Methods and results. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 852–863, 2018.
      Google ScholarLocate open access versionFindings
    • Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Deep image prior. In 2018 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
      Google ScholarLocate open access versionFindings
    • Hien Van Nguyen, Amit Banerjee, and Rama Chellappa. Tracking via object reflectance using a hyperspectral video camera. In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition-Workshops, pages 44–51. IEEE, 2010.
      Google ScholarLocate open access versionFindings
    • Xintao Wang, Ke Yu, Chao Dong, and Chen Change Loy. Recovering realistic texture in image super-resolution by deep spatial feature transform. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 606–615, 2018.
      Google ScholarLocate open access versionFindings
    • Eliot Wycoff, Tsung-Han Chan, Kui Jia, Wing-Kin Ma, and Yi Ma. A non-negative sparse promoting algorithm for high resolution hyperspectral imaging. In Acoustics, Speech and Signal Processing (ICASSP), 2013 IEEE International Conference on, pages 1409–1413. IEEE, 2013.
      Google ScholarLocate open access versionFindings
    • Qi Xie, Minghao Zhou, Qian Zhao, Deyu Meng, Wangmeng Zuo, and Zongben Xu. Multispectral and hyperspectral image fusion by ms/hs fusion net. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1585–1594, 2019.
      Google ScholarLocate open access versionFindings
    • Jingxiang Yang, Yongqiang Zhao, Chen Yi, and Jonathan Cheung-Wai Chan. No-reference hyperspectral image quality assessment via quality-sensitive features learning. Remote Sensing, 9(4):305, 2017.
      Google ScholarLocate open access versionFindings
    • Jingxiang Yang, Yong-Qiang Zhao, and Jonathan Chan. Hyperspectral and multispectral image fusion via deep twobranches convolutional neural network. Remote Sensing, 10(5):800, 2018.
      Google ScholarLocate open access versionFindings
    • Fumihito Yasuma, Tomoo Mitsunaga, Daisuke Iso, and Shree K Nayar. Generalized assorted pixel camera: postcapture control of resolution, dynamic range, and spectrum. IEEE transactions on image processing, 19(9):2241–2253, 2010.
      Google ScholarLocate open access versionFindings
    • Haichao Zhang, Yanning Zhang, Haisen Li, and Thomas S Huang. Generative bayesian image super resolution with natural image prior. IEEE Transactions on Image processing, 21(9):4054–4067, 2012.
      Google ScholarLocate open access versionFindings
    • Lei Zhang, Wei Wei, Chengcheng Bai, Yifan Gao, and Yanning Zhang. Exploiting clustering manifold structure for hyperspectral imagery super-resolution. IEEE Transactions on Image Processing, 27(12):5969–5982, 2018.
      Google ScholarLocate open access versionFindings
    • Lei Zhang, Wei Wei, Yanning Zhang, Chunhua Shen, Anton van den Hengel, and Qinfeng Shi. Cluster sparsity field: An internal hyperspectral imagery prior for reconstruction. International Journal of Computer Vision, 126(8):797–821, 2018.
      Google ScholarLocate open access versionFindings
    • Lei Zhang, Wei Wei, Yanning Zhang, Chunna Tian, and Fei Li. Reweighted laplace prior based hyperspectral compressive sensing for unknown sparsity. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2274–2281, 2015.
      Google ScholarLocate open access versionFindings
    • Maria Zontak and Michal Irani. Internal statistics of a single natural image. In CVPR 2011, pages 977–984. IEEE, 2011.
      Google ScholarLocate open access versionFindings
    Your rating :
    0

     

    Tags
    Comments