Controllable Invariance through Adversarial Feature Learning


Cited by: 76|Bibtex|Views28|Links
variational fair auto-encoderoptimal equilibriumgood generalizationGenerative Adversarial Netscentral moment discrepancyMore(9+)
In the last few years, the dominant paradigm for finding such a representation has shifted from manual feature engineering based on specific domain knowledge to representation learning that is fully data-driven, and often powered by deep neural networks


Learning meaningful representations that maintain the content necessary for a particular task while filtering away detrimental variations is a problem of great interest in machine learning. In this paper, we tackle the problem of learning representations invariant to a specific factor or trait of data. The representation learning process ...More



  • How to produce a data representation that maintains meaningful variations of data while eliminating noisy signals is a consistent theme of machine learning research.
  • The variational fair auto-encoder (VFAE) [Louizos et al, 2016] employs the maximum mean discrepancy (MMD) to eliminate the negative influence of specific “nuisance variables”, such as removing the lighting conditions of images to predict the person’s identity.
  • Given the obtained representation h, the target y is predicted by a predictor M , which effectively models the distribution qM (y | h).
  • The discriminator D is trained to predict s based on the encoded representation h, which effectively maximizes the likelihood qD(s | h).
  • We present the quantitative results showing the superior performance of our proposed framework, and discuss some qualitative analysis which verifies the learned representations have the desired invariance property.
  • Our experiments include three tasks in different domains: (1) fair classification, in which predictions should be unaffected by nuisance factors; (2) language-independent generation which is conducted on the multi-lingual machine translation problem; (3) lighting-independent image classification.
  • Both multi-lingual systems outperform the bilingual encoder-decoder even though multi-lingual systems use similar number of parameters to translate two languages, which shows that learning
  • Without the discriminator, the model’s performance is worse than the standard multi-lingual system, which rules out the possibility that the gain of our model comes from more parameters of separating encoders.
  • Image Classification We report the results in Table 2 with two baselines [Li et al, 2014, Louizos et al, 2016] that use MMD regularizations to remove lighting conditions.
  • In image classification, adversarial training has been shown to able to learn an invariant representation across domains [Ganin and Lempitsky, 2015, Ganin et al, 2016, Bousmalis et al, 2016, Tzeng et al, 2017] and enables classifiers trained on the source domain to be applicable to the target domain.
  • Works targeting the development of fair, bias-free classifiers aim to learn representations invariant to “nuisance variables” that could induce bias and makes the predictions fair, as data-driven models trained using historical data inherit the bias exhibited in the data.
  • Zemel et al [2013] proposes to regularize the 1 distance between representation distributions for data with different nuisance variables to enforce fairness.
  • Louppe et al [2016] propose an adversarial training framework to learn representations independent to a categorical or continuous variable.
  • We cast the representation learning problem as an adversarial game among an encoder, a discriminator, and a predictor.
  • We theoretically analyze the optimal equilibrium of the minimax game and evaluate the performance of our framework on three tasks from different domains empirically.
  • We show that an invariant representation is learned, resulting in better generalization and improvements on the three tasks
  • Shows that the proposed framework induces an invariant representation, and leads to better generalization evidenced by the improved performance
  • Provides theoretical analysis of the equilibrium condition of the minimax game, and give an intuitive interpretation
  • Shows that the proposed approach not only improves upon vanilla discriminative approaches that do not encourage invariance, but outperforms existing approaches that enforce invariant features
  • Focuses mainly on instances where s is a discrete label with multiple choices
Your rating :