Open Compound Domain Adaptation

CVPR, pp. 12403-12412, 2020.

Cited by: 1|Bibtex|Views102|Links
EI
Keywords:
General Research Fundclear distinctiontarget domainunsupervised domain adaptationsemantic segmentationMore(8+)
Weibo:
We propose a novel model which includes a self-organizing curriculum domain adaptation to bootstrap generalization and a memory enhanced feature representation to build agility towards open domains

Abstract:

A typical domain adaptation approach is to adapt models trained on the annotated data in a source domain (e.g., sunny weather) for achieving high performance on the test data in a target domain (e.g., rainy weather). Whether the target contains a single homogeneous domain or multiple heterogeneous domains, existing works always assume tha...More

Code:

Data:

0
Introduction
  • Supervised learning can achieve competitive performance for a visual task when the test data is drawn from the same underlying distribution as the training data.
  • Though domain generalization [24, 22] and latent domain adaptation [16, 11] have attempted to tackle complex target domains, most existing works usually assume that there is a known clear distinction between domains [12, 8, 48, 30, 42]
  • Such a known and clear distinction between domains is hard to define in practice, e.g., test images could be collected in mixed, continually varying, and sometimes never seen weather conditions.
  • With numerous factors jointly contributing to data variance, it becomes implausible to separate data into discrete domains
Highlights
  • Supervised learning can achieve competitive performance for a visual task when the test data is drawn from the same underlying distribution as the training data
  • We propose to study open compound domain adaptation (OCDA), a continuous and more realistic setting for domain
  • We propose a novel approach based on two technical insights into open compound domain adaptation: 1) a curriculum domain adaptation strategy to bootstrap generalization across domain distinction in a data-driven self-organizing fashion and 2) a memory module to increase the model’s agility towards novel domains
  • 2) We develop an open compound domain adaptation solution with two key technical insights: instance-specific curriculum domain adaptation for handling the target of mixed domains and memory augmented features for handling open domains
  • Maximizing the classifier discrepancy increases the robustness to the open domain; it fails to capture the fine-grained latent structure in the compound target domain
  • We propose a novel model which includes a self-organizing curriculum domain adaptation to bootstrap generalization and a memory enhanced feature representation to build agility towards open domains
Methods
  • The authors choose for comparison state-of-the-art methods in both conventional unsupervised domain adaptation (ADDA [48], JAN [30], MCD [42]) and the recent multi-target domain adaptation methods (MTDA [9], BTDA [5], DADA [39]).
  • For the reinforcement learning task, the authors benchmark with MTL, MLP [18] and SynPo [18], a representative work for adaptation across environments.
  • The authors apply these methods to the same backbone networks as ours for a fair comparison
Results
  • ADDA [48] and JAN [30] boost the performance on the compound domain by enforcing global distribution alignment.
  • They sacrifice the performance on the open domain since there is no built-in mechanism for handling any new domains, “overfitting” the model to the seen domains.
  • Compared to other multi-target domain adaptation methods (MTDA [9] and DADA [39]), the approach discovers domain structures and performs domain-aware knowledge transfer, achieving substantial advantages on all the test domains
Conclusion
  • The authors formalize a more realistic topic called open compound domain adaptation for domain-robust learning.
  • The authors propose a novel model which includes a self-organizing curriculum domain adaptation to bootstrap generalization and a memory enhanced feature representation to build agility towards open domains.
  • The authors develop several benchmarks on classification, recognition, segmentation, and reinforcement learning and demonstrate the effectiveness of the model.
Summary
  • Introduction:

    Supervised learning can achieve competitive performance for a visual task when the test data is drawn from the same underlying distribution as the training data.
  • Though domain generalization [24, 22] and latent domain adaptation [16, 11] have attempted to tackle complex target domains, most existing works usually assume that there is a known clear distinction between domains [12, 8, 48, 30, 42]
  • Such a known and clear distinction between domains is hard to define in practice, e.g., test images could be collected in mixed, continually varying, and sometimes never seen weather conditions.
  • With numerous factors jointly contributing to data variance, it becomes implausible to separate data into discrete domains
  • Methods:

    The authors choose for comparison state-of-the-art methods in both conventional unsupervised domain adaptation (ADDA [48], JAN [30], MCD [42]) and the recent multi-target domain adaptation methods (MTDA [9], BTDA [5], DADA [39]).
  • For the reinforcement learning task, the authors benchmark with MTL, MLP [18] and SynPo [18], a representative work for adaptation across environments.
  • The authors apply these methods to the same backbone networks as ours for a fair comparison
  • Results:

    ADDA [48] and JAN [30] boost the performance on the compound domain by enforcing global distribution alignment.
  • They sacrifice the performance on the open domain since there is no built-in mechanism for handling any new domains, “overfitting” the model to the seen domains.
  • Compared to other multi-target domain adaptation methods (MTDA [9] and DADA [39]), the approach discovers domain structures and performs domain-aware knowledge transfer, achieving substantial advantages on all the test domains
  • Conclusion:

    The authors formalize a more realistic topic called open compound domain adaptation for domain-robust learning.
  • The authors propose a novel model which includes a self-organizing curriculum domain adaptation to bootstrap generalization and a memory enhanced feature representation to build agility towards open domains.
  • The authors develop several benchmarks on classification, recognition, segmentation, and reinforcement learning and demonstrate the effectiveness of the model.
Tables
  • Table1: Comparison of domain adaptation settings. Domain Labels tell to which domain each instance belongs. Open Classes refer to novel classes showing up during testing but not training. Open Domains are the domains of which no instances are seen during training
  • Table2: Performance on the C-Digits benchmark. The methods in gray are especially designed for multi-target domain adaptation. †MTDA uses domain labels, while ‡BTDA and DADA use the open domain images during training
  • Table3: Performance on the C-Faces benchmark. The methods in gray are especially designed for multi-target domain adaptation. †MTDA uses domain labels during training
  • Table4: Performance on the C-Driving (left) and C-Mazes benchmarks (right). “SynPo+Aug.” indicates that we equip SynPo with proper color augmentation/randomization during training. Visual illustrations of both datasets are in Figure 6
  • Table5: Summary of notations
  • Table6: Methodology comparisons between MTDA, BTDA, DADA and our OCDA
  • Table7: Statistics of BDD100K dataset in our Open Compound Domain Adaptation (OCDA) setting
  • Table8: The major hyper-parameters used in our experiments. “LR.” stands for learning rate
  • Table9: Per-category IoU(%) results on the C-Driving Benchmark. (BDD100K dataset is used as the real-world target domain data.) The ’train’ and ’bicycle’ categories are not listed because their results are close to zero
Download tables as Excel
Related work
  • We review literature according to Table 1.

    Unsupervised Domain Adaptation. The goal is to retain recognition accuracies in new domains without ground truth annotations [41, 46, 49, 38]. Representative techniques include latent distribution alignment [12], backpropagation [7], gradient reversal [8], adversarial discrimination [48], joint maximum mean discrepancy [30], cycle consistency [17] and maximum classifier discrepancy [42]. While their results are promising, this traditional domain adaptation setting focuses on “one source domain, one target domain”, and cannot deal with more complicated scenarios where multiple target domains are present.

    DComlaasisnLIambaeglses
Funding
  • This research was supported, in part, by the General Research Fund (GRF) of Hong Kong (No 14236516 & No 14203518), Berkeley Deep Drive, DARPA, NSF 1835539, and US Government fund through Etegent Technologies on LowShot Detection in Remote Sensing Imagery
Reference
  • Switzerland handwritten digits dataset. https://github.com/kensanata/numbers. Accessed:201903-15.5, 8
    Findings
  • Andreea Bobu, Eric Tzeng, Judy Hoffman, and Trevor Darrell. Adapting to continuously shifting domains. ICLR Workshop, 2018. 3
    Google ScholarLocate open access versionFindings
  • Zhangjie Cao, Lijia Ma, Mingsheng Long, and Jianmin Wang. Partial adversarial domain adaptation. In ECCV, 2018. 3, 11
    Google ScholarLocate open access versionFindings
  • Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Semantic image segmentation with deep convolutional nets and fully connected crfs. ICLR, 2015. 13
    Google ScholarLocate open access versionFindings
  • Ziliang Chen, Jingyu Zhuang, Xiaodan Liang, and Liang Lin. Blending-target domain adaptation by adversarial metaadaptation networks. In CVPR, 2019. 3, 6, 11, 12
    Google ScholarLocate open access versionFindings
  • Dengxin Dai, Christos Sakaridis, Simon Hecker, and Luc Van Gool. Model adaptation with synthetic and real data for semantic dense foggy scene understanding. IJCV, 2019. 2
    Google ScholarLocate open access versionFindings
  • Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In ICML, 2015. 2, 5, 12
    Google ScholarLocate open access versionFindings
  • Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, Francois Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural networks. JMLR, 2016. 1, 2
    Google ScholarLocate open access versionFindings
  • Behnam Gholami, Pritish Sahu, Ognjen Rudovic, Konstantinos Bousmalis, and Vladimir Pavlovic. Unsupervised multi-target domain adaptation: An information theoretic approach. arXiv preprint arXiv:1810.11547, 2018. 3, 6, 7, 11, 12
    Findings
  • Spyros Gidaris and Nikos Komodakis. Dynamic few-shot visual learning without forgetting. In CVPR, 2018. 5
    Google ScholarLocate open access versionFindings
  • Boqing Gong, Kristen Grauman, and Fei Sha. Reshaping visual datasets for domain adaptation. In NIPS, 2013. 1, 3
    Google ScholarLocate open access versionFindings
  • Boqing Gong, Yuan Shi, Fei Sha, and Kristen Grauman. Geodesic flow kernel for unsupervised domain adaptation. In CVPR, 201, 2
    Google ScholarLocate open access versionFindings
  • Rui Gong, Wen Li, Yuhua Chen, and Luc Van Gool. Dlow: Domain flow for adaptation and generalization. In CVPR, 2019. 3
    Google ScholarLocate open access versionFindings
  • R. Gross, I. Matthews, J. Cohn, T. Kanade, and S. Baker. Multi-pie. In 2008 8th IEEE International Conference on Automatic Face Gesture Recognition, 2008. 5, 12
    Google ScholarLocate open access versionFindings
  • Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016. 6, 11, 13
    Google ScholarLocate open access versionFindings
  • Judy Hoffman, Brian Kulis, Trevor Darrell, and Kate Saenko. Discovering latent domains for multisource domain adaptation. In ECCV, 2012. 1, 3
    Google ScholarLocate open access versionFindings
  • Judy Hoffman, Eric Tzeng, Taesung Park, Jun-Yan Zhu, Phillip Isola, Kate Saenko, Alexei A Efros, and Trevor Darrell. Cycada: Cycle-consistent adversarial domain adaptation. In ICML, 2018. 2, 12, 13
    Google ScholarLocate open access versionFindings
  • Hexiang Hu, Liyu Chen, Boqing Gong, and Fei Sha. Synthesized policies for transfer and adaptation across tasks and environments. In NIPS, 205, 6, 7, 12, 13, 19
    Google ScholarLocate open access versionFindings
  • J. J. Hull. A database for handwritten text recognition research. TPAMI, 1994. 2, 5, 12
    Google ScholarLocate open access versionFindings
  • Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. ICLR, 2015. 13
    Google ScholarLocate open access versionFindings
  • Yann LeCun, Leon Bottou, Yoshua Bengio, Patrick Haffner, et al. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 1998. 2, 5, 6, 12, 13
    Google ScholarLocate open access versionFindings
  • Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. Learning to generalize: Meta-learning for domain generalization. In AAAI, 2018. 1, 3
    Google ScholarLocate open access versionFindings
  • Haoliang Li, Sinno Jialin Pan, Shiqi Wang, and Alex C Kot. Domain generalization with adversarial feature learning. In CVPR, 2018. 3
    Google ScholarLocate open access versionFindings
  • Wen Li, Zheng Xu, Dong Xu, Dengxin Dai, and Luc Van Gool. Domain generalization and adaptation using low rank exemplar svms. TPAMI, 2017. 1
    Google ScholarLocate open access versionFindings
  • Xiaoxiao Li, Ziwei Liu, Ping Luo, Chen Change Loy, and Xiaoou Tang. Not all pixels are equal: Difficulty-aware semantic segmentation via deep layer cascade. In CVPR, 2017. 2
    Google ScholarLocate open access versionFindings
  • Qing Lian, Fengmao Lv, Lixin Duan, and Boqing Gong. Constructing self-motivated pyramid curriculums for crossdomain semantic segmentation: A non-adversarial approach. In ICCV, 2019. 6, 7
    Google ScholarLocate open access versionFindings
  • Weiyang Liu, Yandong Wen, Zhiding Yu, Ming Li, Bhiksha Raj, and Le Song. Sphereface: Deep hypersphere embedding for face recognition. In CVPR, 2017. 5
    Google ScholarLocate open access versionFindings
  • Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, and Stella X Yu. Large-scale long-tailed recognition in an open world. In CVPR, 2019. 5
    Google ScholarLocate open access versionFindings
  • Ziwei Liu, Sijie Yan, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Fashion landmark detection in the wild. In ECCV, 2016. 2
    Google ScholarLocate open access versionFindings
  • Mingsheng Long, Han Zhu, Jianmin Wang, and Michael I Jordan. Deep transfer learning with joint adaptation networks. In ICML, 2017. 1, 2, 6, 7, 8, 14, 17, 18, 20, 21
    Google ScholarLocate open access versionFindings
  • Massimiliano Mancini, Samuel Rota Bulo, Barbara Caputo, and Elisa Ricci. Adagraph: Unifying predictive and continuous domain adaptation through graphs. In CVPR, 2019. 3
    Google ScholarLocate open access versionFindings
  • Massimiliano Mancini, Lorenzo Porzi, Samuel Rota Bulo, Barbara Caputo, and Elisa Ricci. Inferring latent domains for unsupervised deep domain adaptation. TPAMI, 2019. 3
    Google ScholarLocate open access versionFindings
  • Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. NIPS, 2011. 2, 5, 12
    Google ScholarLocate open access versionFindings
  • Sinno Jialin Pan, Ivor W Tsang, James T Kwok, and Qiang Yang. Domain adaptation via transfer component analysis. IEEE Transactions on Neural Networks, 2010. 1
    Google ScholarLocate open access versionFindings
  • Xingang Pan, Ping Luo, Jianping Shi, and Xiaoou Tang. Two at once: Enhancing learning and generalization capacities via ibn-net. In ECCV, 2018. 6, 7, 13, 15
    Google ScholarLocate open access versionFindings
  • Yingwei Pan, Ting Yao, Yehao Li, Yu Wang, Chong-Wah Ngo, and Tao Mei. Transferrable prototypical networks for unsupervised domain adaptation. In CVPR, 2019. 5
    Google ScholarLocate open access versionFindings
  • Pau Panareda Busto and Juergen Gall. Open set domain adaptation. In ICCV, 2017. 3, 11
    Google ScholarLocate open access versionFindings
  • Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In ICCV, 2019. 2
    Google ScholarLocate open access versionFindings
  • Xingchao Peng, Zijun Huang, Ximeng Sun, and Kate Saenko. Domain agnostic learning with disentangled representations. In ICML, 2019. 3, 6, 7, 11, 12
    Google ScholarLocate open access versionFindings
  • Stephan R. Richter, Vibhav Vineet, Stefan Roth, and Vladlen Koltun. Playing for data: Ground truth from computer games. In ECCV, 2016. 5, 12
    Google ScholarLocate open access versionFindings
  • Kate Saenko, Brian Kulis, Mario Fritz, and Trevor Darrell. Adapting visual category models to new domains. In ECCV, 2010. 1, 2
    Google ScholarLocate open access versionFindings
  • Kuniaki Saito, Kohei Watanabe, Yoshitaka Ushiku, and Tatsuya Harada. Maximum classifier discrepancy for unsupervised domain adaptation. In CVPR, 2018. 1, 2, 6, 7, 8, 14, 17, 18, 20, 21
    Google ScholarLocate open access versionFindings
  • Kuniaki Saito, Shohei Yamamoto, Yoshitaka Ushiku, and Tatsuya Harada. Open set domain adaptation by backpropagation. In ECCV, 2018. 3, 11
    Google ScholarLocate open access versionFindings
  • K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015. 6, 11
    Google ScholarLocate open access versionFindings
  • Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In NIPS, 2017. 5
    Google ScholarLocate open access versionFindings
  • Antonio Torralba and Alexei A Efros. Unbiased look at dataset bias. In CVPR, 2011. 1, 2
    Google ScholarLocate open access versionFindings
  • Yi-Hsuan Tsai, Wei-Chih Hung, Samuel Schulter, Kihyuk Sohn, Ming-Hsuan Yang, and Manmohan Chandraker. Learning to adapt structured output space for semantic segmentation. In CVPR, 2018. 6, 7, 13, 14, 15
    Google ScholarLocate open access versionFindings
  • Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation. In CVPR, 2017. 1, 2, 4, 6, 7, 13
    Google ScholarLocate open access versionFindings
  • Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In CVPR, 2017. 2, 14
    Google ScholarLocate open access versionFindings
  • Zuxuan Wu, Xin Wang, Joseph E Gonzalez, Tom Goldstein, and Larry S Davis. Ace: Adapting to changing environments for semantic segmentation. In ICCV, 2019. 3
    Google ScholarLocate open access versionFindings
  • Caiming Xiong, Scott McCloskey, Shao-Hang Hsieh, and Jason J Corso. Latent domains modeling for visual domain adaptation. In AAAI, 2014. 3
    Google ScholarLocate open access versionFindings
  • Zheng Xu, Wen Li, Li Niu, and Dong Xu. Exploiting lowrank structure from latent domains for domain generalization. In ECCV, 2014. 3
    Google ScholarLocate open access versionFindings
  • Fisher Yu, Wenqi Xian, Yingying Chen, Fangchen Liu, Mike Liao, Vashisht Madhavan, and Trevor Darrell. Bdd100k: A diverse driving video database with scalable annotation tooling. arXiv preprint arXiv:1805.04687, 2018. 5, 7, 12
    Findings
  • Huanhuan Yu, Menglei Hu, and Songcan Chen. Multi-target unsupervised domain adaptation without exactly shared categories. arXiv preprint arXiv:1809.00852, 2018. 3
    Findings
  • Jing Zhang, Zewei Ding, Wanqing Li, and Philip Ogunbona. Importance weighted adversarial nets for partial domain adaptation. In CVPR, 2018. 3, 11
    Google ScholarLocate open access versionFindings
  • Yang Zhang, Philip David, Hassan Foroosh, and Boqing Gong. A curriculum domain adaptation approach to the semantic segmentation of urban scenes. TPAMI, 2019. 2, 4
    Google ScholarLocate open access versionFindings
  • Yang Zou, Zhiding Yu, Xiaofeng Liu, BVK Kumar, and Jinsong Wang. Confidence regularized self-training. In ICCV, 2019. 2
    Google ScholarLocate open access versionFindings
  • Yang Zou, Zhiding Yu, BVK Vijaya Kumar, and Jinsong Wang. Unsupervised domain adaptation for semantic segmentation via class-balanced self-training. In ECCV, 2018. 2, 6, 7, 13, 15 (1) In the first stage, we train the network with discriminative centroid loss for 100 epochs as a warm start for the memory-enhanced deep neural network.
    Google ScholarLocate open access versionFindings
  • (2) In the second stage, we further finetune the networks with curriculum sampling and memory modules on top of the backbone network, where a domain adversarial loss [48] is incorporated and the weights learned in the first stage are copied to two identical but independent networks (source network and target network). Only weights of the target network are updated during this stage. Centroids of each class (i.e. constituting elements in the class memory) are calculated in the beginning of this stage, and the classifiers are reinitialized. The model is trained without the discriminative centroid loss in stage 2. Some major hyper-parameters can be found in Table 8.
    Google ScholarFindings
Your rating :
0

 

Tags
Comments