# Consistency Regularization for Generative Adversarial Networks

ICLR, 2020.

EI

Keywords:

Generative Adversarial Networks Consistency Regularization GAN

Weibo:

Abstract:

Generative Adversarial Networks (GANs) are known to be difficult to train, despite considerable research effort. Several regularization techniques for stabilizing training have been proposed, but they introduce non-trivial computational overheads and interact poorly with existing techniques like spectral normalization. In this work, we pr...More

Introduction

- Generative Adversarial Networks (GANs) (Goodfellow et al, 2014) have recently demonstrated impressive results on image-synthesis benchmarks (Radford et al, 2016; Zhang et al, 2017; Miyato & Koyama, 2018; Zhang et al, 2018; Brock et al, 2018; Karras et al, 2019).
- One may anticipate simultaneous regularization and normalization could improve sample quality
- Most of these gradient based regularization methods either provide marginal gains or fail to introduce any improvement when normalization is used (Kurach et al, 2019), which is observed in the experiments.
- These regularization methods and spectral normalization are motivated by controlling Lipschitz constant of the discriminator.
- The authors suspect this might be the reason that applying both does not lead to overlaid gain

Highlights

- Generative Adversarial Networks (GANs) (Goodfellow et al, 2014) have recently demonstrated impressive results on image-synthesis benchmarks (Radford et al, 2016; Zhang et al, 2017; Miyato & Koyama, 2018; Zhang et al, 2018; Brock et al, 2018; Karras et al, 2019)
- Generative Adversarial Networks are composed of two neural networks trained with competing goals: the generator is trained to synthesize realistic samples to fool the discriminator and the discriminator is trained to distinguish real samples from fake ones produced by the generator
- We propose a simple, effective, and computationally cheap method – consistency regularization – to improve the performance of Generative Adversarial Networks
- Consistency regularization is compatible with spectral normalization and results in improvements in all of the many contexts in which we evaluated it
- We have demonstrated consistency regularization is more effective than other regularization methods under different loss functions, neural architectures and optimizer hyper-parameter settings
- We have shown applying consistency regularization on top of state-of-the-art Generative Adversarial Networks models can further greatly boost the performance

Methods

- 2.1 GANS

A GAN consists of a generator network and a discriminator network. The generator G takes a latent variable z ∼ p(z) sampled from a prior distribution and maps it to the observation space X. - In the standard GAN training procedure the generator G and the discriminator D are trained by minimizing the following objectives in an alternating fashion: LD = −Ex∼pdata [log D(x)] − Ez∼p(z) [1 − log D(G(z))] , (1).
- LG = −Ez∼p(z) [log D(G(z))] , where p(z) is usually a standard normal distribution.
- This formulation is originally proposed by Goodfellow et al (2014) as non-saturating (NS) GAN.

Conclusion

- The authors propose a simple, effective, and computationally cheap method – consistency regularization – to improve the performance of GANs.
- The authors have demonstrated consistency regularization is more effective than other regularization methods under different loss functions, neural architectures and optimizer hyper-parameter settings.
- The authors have shown applying consistency regularization on top of state-of-the-art GAN models can further greatly boost the performance.
- The authors have conducted a thorough study on the design choices and hyper-parameters of consistency regularization

Summary

## Introduction:

Generative Adversarial Networks (GANs) (Goodfellow et al, 2014) have recently demonstrated impressive results on image-synthesis benchmarks (Radford et al, 2016; Zhang et al, 2017; Miyato & Koyama, 2018; Zhang et al, 2018; Brock et al, 2018; Karras et al, 2019).- One may anticipate simultaneous regularization and normalization could improve sample quality
- Most of these gradient based regularization methods either provide marginal gains or fail to introduce any improvement when normalization is used (Kurach et al, 2019), which is observed in the experiments.
- These regularization methods and spectral normalization are motivated by controlling Lipschitz constant of the discriminator.
- The authors suspect this might be the reason that applying both does not lead to overlaid gain
## Methods:

2.1 GANS

A GAN consists of a generator network and a discriminator network. The generator G takes a latent variable z ∼ p(z) sampled from a prior distribution and maps it to the observation space X.- In the standard GAN training procedure the generator G and the discriminator D are trained by minimizing the following objectives in an alternating fashion: LD = −Ex∼pdata [log D(x)] − Ez∼p(z) [1 − log D(G(z))] , (1).
- LG = −Ez∼p(z) [log D(G(z))] , where p(z) is usually a standard normal distribution.
- This formulation is originally proposed by Goodfellow et al (2014) as non-saturating (NS) GAN.
## Conclusion:

The authors propose a simple, effective, and computationally cheap method – consistency regularization – to improve the performance of GANs.- The authors have demonstrated consistency regularization is more effective than other regularization methods under different loss functions, neural architectures and optimizer hyper-parameter settings.
- The authors have shown applying consistency regularization on top of state-of-the-art GAN models can further greatly boost the performance.
- The authors have conducted a thorough study on the design choices and hyper-parameters of consistency regularization

- Table1: Best FID scores for unconditional image generation on CIFAR-10 and CelebA
- Table2: Comparison of our technique with state-of-the-art GAN models including SNGAN (Miyato & Koyama, 2018), SAGAN (Zhang et al, 2019) and BigGAN (<a class="ref-link" id="cBrock_et+al_2018_a" href="#rBrock_et+al_2018_a">Brock et al, 2018</a>) for class conditional image generation on CIFAR-10 and ImageNet in terms of FID. BigGAN is the BigGAN implementation of <a class="ref-link" id="cKurach_et+al_2019_a" href="#rKurach_et+al_2019_a">Kurach et al (2019</a>). CR-BigGAN has the exactly same architecture as BigGAN and is trained with the same settings. The only difference is CR-BigGAN adds consistency regularization
- Table3: FID scores on CIFAR-10 for different types of image augmentation. Gaussian noise is the worst, and random shift and flip is the best, consistent with general consensus on the best way to perform image optimization on CIFAR-10 (Zagoruyko & Komodakis, 2016)
- Table4: Hyper-parameters of the optimizer used in our experiments
- Table5: Training speed of discriminator updates for SNDCGAN on CIFAR-10
- Table6: Best Inception Score for unconditional image generation on CIFAR-10

Funding

- Effective training stabilizer based on the notion of consistency regularization—a popular technique in the semi-supervised learning literature
- Shows that it works across a large range of GAN variants and datasets
- Shows that applying this technique on top of existing GAN models leads to new state-of-the-art results as measured by Frechet Inception Distance
- Shows that applying the proposed technique can further boost the performance of state-of-the-art GAN models
- Proposes a consistency regularization on the GAN discriminator during training

Reference

- Martín Arjovsky and Léon Bottou. Towards principled methods for training generative adversarial networks. In ICLR, 2017.
- Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein GAN. arXiv:1701.07875, 2017.
- Philip Bachman, Ouais Alsharif, and Doina Precup. Learning with pseudo-ensembles. In NeurIPS, 2014.
- David Berthelot, Nicholas Carlini, Ian J. Goodfellow, Nicolas Papernot, Avital Oliver, and Colin Raffel. Mixmatch: A holistic approach to semi-supervised learning. arXiv:1905.02249, 2019.
- Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv:1809.11096, 2018.
- Ting Chen, Xiaohua Zhai, Marvin Ritter, Mario Lucic, and Neil Houlsby. Self-supervised gans via auxiliary rotation loss. In CVPR, 2019.
- Terrance DeVries and Graham W Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv:1708.04552, 2017.
- Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. Generative adversarial nets. In NeurIPS, 2014.
- Ishaan Gulrajani, Faruk Ahmed, Martín Arjovsky, Vincent Dumoulin, and Aaron C. Courville. Improved training of wasserstein GANs. In NeurIPS, 2017.
- Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
- Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In NeurIPS, 2017.
- Weihua Hu, Takeru Miyato, Seiya Tokui, Eiichi Matsumoto, and Masashi Sugiyama. Learning discrete representations via information maximizing self-augmented training. In ICML, 2017.
- Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of GANs for improved quality, stability, and variation. In ICLR, 2018.
- Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In CVPR, 2019.
- Naveen Kodali, Jacob Abernethy, James Hays, and Zsolt Kira. On convergence and stability of gans. arXiv preprint arXiv:1705.07215, 2017.
- Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
- Karol Kurach, Mario Lucic, Xiaohua Zhai, Marcin Michalski, and Sylvain Gelly. A large-scale study on regularization and normalization in gans. In ICML, 2019.
- Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning. arXiv preprint arXiv:1610.02242, 2016.
- Jae Hyun Lim and Jong Chul Ye. Geometric GAN. arXiv:1705.02894, 2017.
- Mario Lucic, Karol Kurach, Marcin Michalski, Sylvain Gelly, and Olivier Bousquet. Are gans created equal? A large-scale study. In NeurIPS, 2018.

Tags

Comments