Controllable Invariance through Adversarial Feature Learning
ADVANCES IN NEURAL INFORMATION PROCESSING SYSTEMS 30 (NIPS 2017), pp. 585-596, 2017.
variational fair auto-encoderoptimal equilibriumgood generalizationGenerative Adversarial Netscentral moment discrepancyMore(9+)
In the last few years, the dominant paradigm for finding such a representation has shifted from manual feature engineering based on specific domain knowledge to representation learning that is fully data-driven, and often powered by deep neural networks
Learning meaningful representations that maintain the content necessary for a particular task while filtering away detrimental variations is a problem of great interest in machine learning. In this paper, we tackle the problem of learning representations invariant to a specific factor or trait of data. The representation learning process ...More
PPT (Upload PPT)
- How to produce a data representation that maintains meaningful variations of data while eliminating noisy signals is a consistent theme of machine learning research.
- The variational fair auto-encoder (VFAE) [Louizos et al, 2016] employs the maximum mean discrepancy (MMD) to eliminate the negative influence of specific “nuisance variables”, such as removing the lighting conditions of images to predict the person’s identity.
- Given the obtained representation h, the target y is predicted by a predictor M , which effectively models the distribution qM (y | h).
- The discriminator D is trained to predict s based on the encoded representation h, which effectively maximizes the likelihood qD(s | h).
- We present the quantitative results showing the superior performance of our proposed framework, and discuss some qualitative analysis which verifies the learned representations have the desired invariance property.
- Our experiments include three tasks in different domains: (1) fair classification, in which predictions should be unaffected by nuisance factors; (2) language-independent generation which is conducted on the multi-lingual machine translation problem; (3) lighting-independent image classification.
- Both multi-lingual systems outperform the bilingual encoder-decoder even though multi-lingual systems use similar number of parameters to translate two languages, which shows that learning
- Without the discriminator, the model’s performance is worse than the standard multi-lingual system, which rules out the possibility that the gain of our model comes from more parameters of separating encoders.
- Image Classification We report the results in Table 2 with two baselines [Li et al, 2014, Louizos et al, 2016] that use MMD regularizations to remove lighting conditions.
- In image classification, adversarial training has been shown to able to learn an invariant representation across domains [Ganin and Lempitsky, 2015, Ganin et al, 2016, Bousmalis et al, 2016, Tzeng et al, 2017] and enables classifiers trained on the source domain to be applicable to the target domain.
- Works targeting the development of fair, bias-free classifiers aim to learn representations invariant to “nuisance variables” that could induce bias and makes the predictions fair, as data-driven models trained using historical data inherit the bias exhibited in the data.
- Zemel et al  proposes to regularize the 1 distance between representation distributions for data with different nuisance variables to enforce fairness.
- Louppe et al  propose an adversarial training framework to learn representations independent to a categorical or continuous variable.
- We cast the representation learning problem as an adversarial game among an encoder, a discriminator, and a predictor.
- We theoretically analyze the optimal equilibrium of the minimax game and evaluate the performance of our framework on three tasks from different domains empirically.
- We show that an invariant representation is learned, resulting in better generalization and improvements on the three tasks
- Shows that the proposed framework induces an invariant representation, and leads to better generalization evidenced by the improved performance
- Provides theoretical analysis of the equilibrium condition of the minimax game, and give an intuitive interpretation
- Shows that the proposed approach not only improves upon vanilla discriminative approaches that do not encourage invariance, but outperforms existing approaches that enforce invariant features
- Focuses mainly on instances where s is a discrete label with multiple choices