Towards Automated Semi-Supervised Learning

national conference on artificial intelligence, 2019.

Cited by: 4|Bibtex|Views262|Links
EI
Keywords:
AUTOhuman interventionhigh qualityunlabeled datumssl techniqueMore(16+)
Weibo:
Automated Machine Learning attempts to build an appropriate machine learning model for unseen data set in an automatic manner

Abstract:

Automated Machine Learning (AutoML) aims to build an appropriate machine learning model for any unseen dataset automatically, i.e., without human intervention. Great efforts have been devoted on AutoML while they typically focus on supervised learning. In many applications, however, semisupervised learning (SSL) are widespread and current...More

Code:

Data:

Introduction
  • Given a dataset, a finetuned learning model is built by human.
  • Google Cloud AutoML is a suite of machine learning products that can automatically train high quality models by leveraging google’s state-of-the-art transfer learning techniques and neural architecture search techniques.3
  • These above studies show that one could automatically select a quite good learning model and hyperparameter for supervised learning problems
  • Google Cloud AutoML is a suite of machine learning products that can automatically train high quality models by leveraging google’s state-of-the-art transfer learning techniques and neural architecture search techniques.3 These above studies show that one could automatically select a quite good learning model and hyperparameter for supervised learning problems
Highlights
  • In traditional machine learning, given a dataset, a finetuned learning model is built by human
  • Google Cloud Automated Machine Learning is a suite of machine learning products that can automatically train high quality models by leveraging google’s state-of-the-art transfer learning techniques and neural architecture search techniques.3
  • In this work we study automated supervised learning and it is notable that existing Automated Machine Learning techniques could not directly be applied for the automated supervised learning problem, since supervised learning introduces some new challenges
  • Extensive empirical results on 40 datasets over 200 cases demonstrate that our proposal achieves highly competitive or better performance compared to the state-of-the-art Automated Machine Learning system AUTO-SKLEARN and classical supervised learning techniques, in addition unlike classical supervised learning techniques often degenerate performance significantly, our proposal seldom suffers from such deficiency
  • Automated Machine Learning (AutoML) attempts to build an appropriate machine learning model for unseen data set in an automatic manner
  • Extensive empirical results show that our proposal outperforms classical supervised learning techniques and state-of-the-art Automated Machine Learning system AUTO-SKLEARN, in addition clearly improves the reliability of supervised learning
Methods
  • Number of labeled instances

    TSVM 12/15/13 13/11/16 13/10/17 12/11/17 13/11/16

    ACC CMN 10/16/14 13/9/18 13/9/18 11/10/19 12/9/19

    ASSL 11/25/4 13/22/5 13/24/3 10/29/1 11/26/3

    TSVM 10/14/16 11/13/16 9/19/12 12/13/15 12/16/12

    AUC CMN 9/14/17 9/11/20 9/11/20 8/12/20 10/10/20

    ASSL 8/24/8 8/27/5 9/28/3 10/26/4 9/28/3 exact appropriate features for data sets by not only traditional meta-features but unsupervised learning.
  • ACC CMN 10/16/14 13/9/18 13/9/18 11/10/19 12/9/19.
  • AUC CMN 9/14/17 9/11/20 9/11/20 8/12/20 10/10/20.
  • ASSL 8/24/8 8/27/5 9/28/3 10/26/4 9/28/3 exact appropriate features for data sets by not only traditional meta-features but unsupervised learning.
  • Extensive empirical results show that the proposal outperforms classical SSL techniques and state-of-the-art AutoML system AUTO-SKLEARN, in addition clearly improves the reliability of SSL.
  • The authors' system could be improved by removing some shortcomings.
  • The authors have not yet tackled SSL with multi-class problems or very large-scale datasets.
  • Applying the system to deep learning models and deriving a good representation of datasets would be a worth-studying future work
Conclusion
  • Automated Machine Learning (AutoML) attempts to build an appropriate machine learning model for unseen data set in an automatic manner.
Summary
  • Introduction:

    Given a dataset, a finetuned learning model is built by human.
  • Google Cloud AutoML is a suite of machine learning products that can automatically train high quality models by leveraging google’s state-of-the-art transfer learning techniques and neural architecture search techniques.3
  • These above studies show that one could automatically select a quite good learning model and hyperparameter for supervised learning problems
  • Google Cloud AutoML is a suite of machine learning products that can automatically train high quality models by leveraging google’s state-of-the-art transfer learning techniques and neural architecture search techniques.3 These above studies show that one could automatically select a quite good learning model and hyperparameter for supervised learning problems
  • Methods:

    Number of labeled instances

    TSVM 12/15/13 13/11/16 13/10/17 12/11/17 13/11/16

    ACC CMN 10/16/14 13/9/18 13/9/18 11/10/19 12/9/19

    ASSL 11/25/4 13/22/5 13/24/3 10/29/1 11/26/3

    TSVM 10/14/16 11/13/16 9/19/12 12/13/15 12/16/12

    AUC CMN 9/14/17 9/11/20 9/11/20 8/12/20 10/10/20

    ASSL 8/24/8 8/27/5 9/28/3 10/26/4 9/28/3 exact appropriate features for data sets by not only traditional meta-features but unsupervised learning.
  • ACC CMN 10/16/14 13/9/18 13/9/18 11/10/19 12/9/19.
  • AUC CMN 9/14/17 9/11/20 9/11/20 8/12/20 10/10/20.
  • ASSL 8/24/8 8/27/5 9/28/3 10/26/4 9/28/3 exact appropriate features for data sets by not only traditional meta-features but unsupervised learning.
  • Extensive empirical results show that the proposal outperforms classical SSL techniques and state-of-the-art AutoML system AUTO-SKLEARN, in addition clearly improves the reliability of SSL.
  • The authors' system could be improved by removing some shortcomings.
  • The authors have not yet tackled SSL with multi-class problems or very large-scale datasets.
  • Applying the system to deep learning models and deriving a good representation of datasets would be a worth-studying future work
  • Conclusion:

    Automated Machine Learning (AutoML) attempts to build an appropriate machine learning model for unseen data set in an automatic manner.
Tables
  • Table1: List of Meta-Features in AUTO-SSL
  • Table2: Accuracy % (mean ± std) on 20 labeled instances for the compared methods. For the compared methods, if the performance is significantly better/worse than the baseline
  • Table3: AUC % (mean ± std) on 20 labeled instances for the compared methods. For the compared methods, if the performance is significantly better/worse than the baseline
  • Table4: The counts of Win/Tie/Loss against SVM with respect to all compared methods. ‘Win/Tie/Loss’ counts the datasets for which the compared methods is statistically significantly better/comparable/significantly worse than SVM (paired t-tests at 95% significance level). The method with the smallest number of losses against SVM is bolded
Download tables as Excel
Related work
  • This work is mostly related to two branches of studies. AutoML: AutoML is challenging. Many issues have been raised (Guyon et al 2016), such as automated feature engineering (Guyon et al 2016), automated model selection (Sun 2016), automated hyperparameter optimization (Hutter, Hoos, and Leyton-Brown 2011). From the systematical scheme aspect, one of the earliest work for AutoML is AUTOWEKA (Hall et al 2009) which combines the machine learning framework WEKA with a bayesian optimization method to select a good configuration for a new dataset. Later on, to further alleviate the high computational cost and derive a more accurate solution, AUTO-SKLEARN (Feurer et al 2015) improves AUTO-WEKA and uses meta-learning (Lemke, Budka, and Gabrys 2015) step to warmstart the bayesian optimization procedure, and finally includes an automated ensemble construction for robustness. Although many AutoML techniques have been proposed, they typically work on supervised learning, while the efforts on semi-supervised learning (SSL) remain to be limited.
Funding
  • This research was supported by the National Key R&D Program of China (2018YFB1004301) and the National Natural Science Foundation of China (61772262)
Reference
  • Bardenet, R.; Brendel, M.; Kegl, B.; and Sebag, M. 2013. Collaborative hyperparameter tuning. In Proceedings of the 30th International Conference on Machine Learning, 199– 207.
    Google ScholarLocate open access versionFindings
  • Brazdil, P.; Carrier, C. G.; Soares, C.; and Vilalta, R. 2008. Metalearning: Applications to data mining. Springer Science & Business Media.
    Google ScholarFindings
  • Chapelle, O.; Scholkopf, B.; and Zien, A. 2006. SemiSupervised Learning. Cambridge, MA: MIT Press.
    Google ScholarFindings
  • Davies, D. L., and Bouldin, D. W. 1979. A cluster separation measure. IEEE Transactions on Pattern Analysis and Machine Intelligence (2):224–227.
    Google ScholarLocate open access versionFindings
  • Feurer, M.; Klein, A.; Eggensperger, K.; Springenberg, J. T.; Blum, M.; and Hutter, F. 201Efficient and robust automated machine learning. In Advances in Neural Information Processing Systems 28. 2962–2970.
    Google ScholarLocate open access versionFindings
  • Feurer, M.; Springenberg, J. T.; and Hutter, F. 2015. Initializing bayesian hyperparameter optimization via meta-learning. In Proceedings of the 29th AAAI Conference on Artificial Intelligence, 1128–1135.
    Google ScholarLocate open access versionFindings
  • Guyon, I.; Chaabane, I.; Escalante, H.; and et al. 2016. A brief review of the chalearn automl challenge. In Proceedings of AutoML workshop on the 33rd International Conference on Machine Learning.
    Google ScholarLocate open access versionFindings
  • Hall, M.; Frank, E.; Holmes, G.; Pfahringer, B.; Reutemann, P.; and Witten, I. H. 2009. The weka data mining software: an update. ACM SIGKDD Explorations Newsletter 11(1):10–18.
    Google ScholarLocate open access versionFindings
  • Hutter, F.; Hoos, H. H.; and Leyton-Brown, K. 2011. Sequential model-based optimization for general algorithm configuration. In Proceedings of International Conference on Learning and Intelligent Optimization, 507–523.
    Google ScholarLocate open access versionFindings
  • Jain, A. K., and Dubes, R. C. 1988. Algorithms for clustering data. Prentice-Hall, Inc.
    Google ScholarFindings
  • Jain, A. K. 2010. Data clustering: 50 years beyond k-means. Pattern Recognition Letters 31(8):651–666.
    Google ScholarLocate open access versionFindings
  • Joachims, T. 1999. Transductive inference for text classification using support vector machines. In Proceedings of the 16th International Conference on Machine Learning, 200–209.
    Google ScholarLocate open access versionFindings
  • Kohavi, R. 1995. A study of cross-validation and bootstrap for accuracy estimation and model selection. In Proceedings of the 9th AAAI Conference on Artificial Intelligence, volume 14, 1137–1145.
    Google ScholarLocate open access versionFindings
  • Krijthe, J. H., and Loog, M. 2015. Implicitly constrained semi-supervised least squares classification. In Advances in 14th International Symposium on Intelligent Data Analysis, 158–169.
    Google ScholarLocate open access versionFindings
  • Lemke, C.; Budka, M.; and Gabrys, B. 20Metalearning: a survey of trends and technologies. Artificial Intelligence Review 44(1):117–130.
    Google ScholarLocate open access versionFindings
  • Li, Y.-F., and Zhou, Z.-H. 2015. Towards making unlabeled data never hurt. IEEE Transactions on Pattern Analysis and Machine Intelligence 37(1):175–188.
    Google ScholarLocate open access versionFindings
  • Li, Y.-F.; Kwok, J.; and Zhou, Z.-H. 2016. Towards safe semisupervised learning for multivariate performance measures. In Proceedings of the 30th AAAI Conference on Artificial Intelligence, 1816–1822.
    Google ScholarLocate open access versionFindings
  • Li, Y.-F.; Wang, S.-B.; and Zhou, Z.-H. 2016. Graph quality judgement: a large margin expedition. In Proceedings of the 25th International Joint Conference on Artificial Intelligence, 1725–1731.
    Google ScholarLocate open access versionFindings
  • Li, Y.-F.; Zha, H.-W.; and Zhou, Z.-H. 2017. Learning safe prediction for semi-supervised regression. In Proceedings of the 31st AAAI Conference on Artificial Intelligence, 2217– 2223.
    Google ScholarLocate open access versionFindings
  • Maulik, U., and Bandyopadhyay, S. 2002. Performance evaluation of some clustering algorithms and validity indices. IEEE Transactions on Pattern Analysis and Machine Intelligence 24(12):1650–1654.
    Google ScholarLocate open access versionFindings
  • Michie, D.; Spiegelhalter, D. J.; Taylor, C. C.; and Campbell, J. 1994. Machine Learning, Neural and Statistical Classification. Ellis Horwood.
    Google ScholarLocate open access versionFindings
  • Sun, L. 2016. Automl challenge: System description. In Proceedings of AutoML workshop on the 33rd International Conference on Machine Learning.
    Google ScholarLocate open access versionFindings
  • Thornton, C.; Hutter, F.; Hoos, H. H.; and Leyton-Brown, K. 2013. Auto-weka: Combined selection and hyperparameter optimization of classification algorithms. In Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data mining, 847–855.
    Google ScholarLocate open access versionFindings
  • Vapnik, V. N. 1999. An overview of statistical learning theory. IEEE Transactions on Neural Networks 10(5):988–999.
    Google ScholarLocate open access versionFindings
  • Von Luxburg, U. 2007. A tutorial on spectral clustering. Statistics and Computing 17(4):395–416.
    Google ScholarLocate open access versionFindings
  • Zhou, Z.-H., and Li, M. 2010. Semi-supervised learning by disagreement. Knowledge and Information Systems 24(3):415–439.
    Google ScholarLocate open access versionFindings
  • Zhou, D.; Bousquet, O.; Lal, T. N.; Weston, J.; and Scholkopf, B. 2004. Learning with local and global consistency. In Advances in Neural Information Processing Systems 16. 321– 328.
    Google ScholarLocate open access versionFindings
  • Zhu, X.; Ghahramani, Z.; and Lafferty, J. D. 2003. Semisupervised learning using gaussian fields and harmonic functions. In Proceedings of the 20th International Conference on Machine Learning, 912–919.
    Google ScholarLocate open access versionFindings
  • Zhu, X. 2008. Semi-supervised learning literature survey. Technical report, University of Wisconsin–Madison, Madison, WI.
    Google ScholarFindings
Your rating :
0

 

Tags
Comments