MoGA: Searching Beyond MobileNetV3

ICASSP, pp. 4042-4046, 2019.

Cited by: 0|Bibtex|Views93|Links
EI
Keywords:
squeeze and excitation moduleneural architecture searchneural networkarchitecture searchmoga bMore(9+)
Weibo:
Our total search cost has been substantially reduced to 12 GPU days

Abstract:

The evolution of MobileNets has laid a solid foundation for neural network application on the mobile end. With the latest MobileNetV3, neural architecture search again claimed its supremacy on network design. Till today all mobile methods mainly focus on CPU latency instead of GPU, the latter, however, has lower overhead and interferenc...More

Code:

Data:

0
Introduction
  • The MobileNets trilogy has opened a gate to on-device artificial intelligence for the mobile vision world (Howard et al 2017; Sandler et al 2018; Howard et al 2019).
  • The guideline in designing mobile architecture is that should the high performance be concerned, and the authors must strive for low latency in favor of rapid responsiveness and improved power efficiency to prolong battery life.
  • The authors aim to bring forward the frontier of mobile neural architecture design by stretching out the repre-.
Highlights
  • The MobileNets trilogy has opened a gate to on-device artificial intelligence for the mobile vision world (Howard et al 2017; Sandler et al 2018; Howard et al 2019)
  • Our total search cost has been substantially reduced to 12 GPU days
  • The trained supernet is once-for-all since the same supernet caters for all mobile contexts
  • It requires o(1) search cost when applying to a new mobile device
  • We employ an automated search approach in the search space adapted from MnasNet and MobileNetV3, which generates a new set of state-of-theart architectures for mobile settings
  • MoGAC hits 75.3% top-1 ImageNet accuracy, which outperforms MobileNetV3 with competing mobile GPU latency at similar FLOPs and an equal number of parameters
Results
  • Model Selection For the chosen weighted NSGA-II equipped with hierarchical mutation (Chu et al 2019b), we.
  • 16*112*112 MBE6 K5 24*56*56 MBE6 K7.
  • 40*28*28 MBE3 K3 SE 40*28*28 MBE6 K3 80*14*14 MBE6 K3
Conclusion
  • The authors have discussed several critical issues in mobile neural architecture design.
  • The authors adopt weighted fitness strategy to comfort more valuable objectives like accuracy and latency, other than the number of parameters.
  • The trained supernet is once-for-all since the same supernet caters for all mobile contexts.
  • It requires o(1) search cost when applying to a new mobile device.
  • MoGAC hits 75.3% top-1 ImageNet accuracy, which outperforms MobileNetV3 with competing mobile GPU latency at similar FLOPs and an equal number of parameters
Summary
  • Introduction:

    The MobileNets trilogy has opened a gate to on-device artificial intelligence for the mobile vision world (Howard et al 2017; Sandler et al 2018; Howard et al 2019).
  • The guideline in designing mobile architecture is that should the high performance be concerned, and the authors must strive for low latency in favor of rapid responsiveness and improved power efficiency to prolong battery life.
  • The authors aim to bring forward the frontier of mobile neural architecture design by stretching out the repre-.
  • Objectives:

    The authors aim to bring forward the frontier of mobile neural architecture design by stretching out the repre-.
  • Results:

    Model Selection For the chosen weighted NSGA-II equipped with hierarchical mutation (Chu et al 2019b), we.
  • 16*112*112 MBE6 K5 24*56*56 MBE6 K7.
  • 40*28*28 MBE3 K3 SE 40*28*28 MBE6 K3 80*14*14 MBE6 K3
  • Conclusion:

    The authors have discussed several critical issues in mobile neural architecture design.
  • The authors adopt weighted fitness strategy to comfort more valuable objectives like accuracy and latency, other than the number of parameters.
  • The trained supernet is once-for-all since the same supernet caters for all mobile contexts.
  • It requires o(1) search cost when applying to a new mobile device.
  • MoGAC hits 75.3% top-1 ImageNet accuracy, which outperforms MobileNetV3 with competing mobile GPU latency at similar FLOPs and an equal number of parameters
Tables
  • Table1: Each layer in our search space has 12 choices. SE: Squeeze-and-Excitation
  • Table2: The architecture of MoGA-A. Note t, c, s refer to expansion rate, output channel size and stride respectively. SE for squeeze-and-excitation, NL for non-linearity. k for the number of categories
  • Table3: Comparison of mobile models on ImageNet. : Our reimplementation. Numbers within the parentheses are reported by its authors, same for below. †: Based on its published code. ‡: Samsung Galaxy S8. ∗: Samsung Note8
Download tables as Excel
Related work
  • During the era of human craftsmanship, MobileNetV1 and V2 (Howard et al 2017; Sandler et al 2018) have widely disseminated depthwise separable convolutions and inverted residuals with linear bottlenecks. Moreover, Squeeze and excitation blocks are later introduced in (Hu, Shen, and Sun 2018) to enrich residual modules from ResNet (He et al 2016).

    In their aftermath, a series of automated architectures are searched based on these building blocks (Tan et al 2019; Cai, Zhu, and Han 2019; Chu et al 2019a; Howard et al 2019). For instance, MnasNet frames a factorized hierarchical search space with MobileNetV2’s inverted bottleneck convolution blocks (MB) of variable kernel sizes and expansion rates. Its latest variation also includes an option of squeeze and excitation module (SE) (Tan et al 2019). ProxylessNAS and FairNAS adopt a similar design (Cai, Zhu, and Han 2019; Chu et al 2019a) without SE modules, while MobileNetV3 achieves a new state of the art by integrating SE within MnasNet search space, along with numerous techniques like Platform-Aware NAS (Tan et al 2019), NetAdapt (Yang et al 2018) and improved non-linearities (Howard et al 2019).
Reference
  • [Abadi et al. 2015] Abadi, M.; Agarwal, A.; Barham, P.; Brevdo, E.; Chen, Z.; Citro, C.; Corrado, G. S.; Davis, A.; Dean, J.; Devin, M.; Ghemawat, S.; Goodfellow, I.; Harp, A.; Irving, G.; Isard, M.; Jia, Y.; Jozefowicz, R.; Kaiser, L.; Kudlur, M.; Levenberg, J.; Mane, D.; Monga, R.; Moore, S.; Murray, D.; Olah, C.; Schuster, M.; Shlens, J.; Steiner, B.; Sutskever, I.; Talwar, K.; Tucker, P.; Vanhoucke, V.; Vasudevan, V.; Viegas, F.; Vinyals, O.; Warden, P.; Wattenberg, M.; Wicke, M.; Yu, Y.; and Zheng, X. 2015. TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems. tag: v1.14.0-rc0. Software available from tensorflow.org.
    Google ScholarFindings
  • [Bender et al. 2018] Bender, G.; Kindermans, P.-J.; Zoph, B.; Vasudevan, V.; and Le, Q. 2018. Understanding and Simplifying One-Shot Architecture Search. In International Conference on Machine Learning, 549–558.
    Google ScholarLocate open access versionFindings
  • [Cai, Gan, and Han 2019] Cai, H.; Gan, C.; and Han, S. 2019. Once for All: Train One Network and Specialize it for Efficient Deployment. arXiv preprint. arXiv:1908.09791.
    Findings
  • [Cai, Zhu, and Han 2019] Cai, H.; Zhu, L.; and Han, S. 2019. ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware. In International Conference on Learning Representations.
    Google ScholarLocate open access versionFindings
  • [Chu et al. 2019a] Chu, X.; Zhang, B.; Xu, R.; and Li, J. 2019a. FairNAS: Rethinking Evaluation Fairness of Weight Sharing Neural Architecture Search. arXiv preprint. arXiv:1907.01845.
    Findings
  • [Chu et al. 2019b] Chu, X.; Zhang, B.; Xu, R.; and Ma, H. 2019b. Multi-Objective Reinforced Evolution in Mobile Neural Architecture Search. arXiv preprint. arXiv:1901.01074.
    Findings
  • [Deb et al. 2002] Deb, K.; Pratap, A.; Agarwal, S.; and Meyarivan, T. 2002. A Fast and Elitist Multiobjective Genetic Algorithm: NSGA-II. IEEE Transactions on Evolutionary Computation 6(2):182–197.
    Google ScholarLocate open access versionFindings
  • [Deng et al. 2009] Deng, J.; Dong, W.; Socher, R.; Li, L.-J.; Li, K.; and Fei-Fei, L. 2009. ImageNet: A Large-Scale Hierarchical Image Database. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 248– 255. IEEE.
    Google ScholarLocate open access versionFindings
  • [Dong et al. 2018] Dong, J.-D.; Cheng, A.-C.; Juan, D.-C.; Wei, W.; and Sun, M. 2018. DPP-Net: Device-aware Progressive Search for Pareto-optimal Neural Architectures. In Proceedings of the European Conference on Computer Vision, 517–531.
    Google ScholarLocate open access versionFindings
  • [Goyal et al. 2017] Goyal, P.; Dollar, P.; Girshick, R.; Noordhuis, P.; Wesolowski, L.; Kyrola, A.; Tulloch, A.; Jia, Y.; and He, K. 2017.
    Google ScholarFindings
  • [Guo et al. 2019] Guo, Z.; Zhang, X.; Mu, H.; Heng, W.; Liu, Z.; Wei, Y.; and Sun, J. 2019. Single Path One-Shot Neural
    Google ScholarFindings
  • [He et al. 2016] He, K.; Zhang, X.; Ren, S.; and Sun, J. 2016. Deep Residual Learning for Image Recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 770–778.
    Google ScholarLocate open access versionFindings
  • [Howard et al. 2017] Howard, A. G.; Zhu, M.; Chen, B.; Kalenichenko, D.; Wang, W.; Weyand, T.; Andreetto, M.; and Adam, H. 2017. MobileNets: Efficient. Convolutional Neural Networks for Mobile Vision Applications. arXiv preprint. arXiv:1704.04861.
    Findings
  • [Howard et al. 2019] Howard, A.; Sandler, M.; Chu, G.; Chen, L.-C.; Chen, B.; Tan, M.; Wang, W.; Zhu, Y.; Pang, R.; Vasudevan, V.; et al. 2019. Searching for MobileNetV3. arXiv preprint. arXiv:1905.02244.
    Findings
  • [Hu, Shen, and Sun 2018] Hu, J.; Shen, L.; and Sun, G. 2018. Squeeze-and-Excitation Networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 7132–7141.
    Google ScholarLocate open access versionFindings
  • [Liu, Simonyan, and Yang 2019] Liu, H.; Simonyan, K.; and Yang, Y. 2019. DARTS: Differentiable Architecture Search. In International Conference on Learning Representations.
    Google ScholarLocate open access versionFindings
  • [Qualcomm 2019] Qualcomm. 2019. Snapdragon Neural Processing Engine SDK. https://developer.qualcomm.com/software/qualcomm-neural-processing-sdk, version:1.27.1.382.
    Locate open access versionFindings
  • [Sandler et al. 2018] Sandler, M.; Howard, A.; Zhu, M.; Zhmoginov, A.; and Chen, L.-C. 20MobileNetV2: Inverted Residuals and Linear Bottlenecks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 4510–4520.
    Google ScholarLocate open access versionFindings
  • [Srivastava et al. 2014] Srivastava, N.; Hinton, G.; Krizhevsky, A.; Sutskever, I.; and Salakhutdinov, R. 2014. Dropout: A Simple Way to Prevent Neural Networks from Overfitting. The Journal of Machine Learning Research 15(1):1929–1958.
    Google ScholarLocate open access versionFindings
  • [Stamoulis et al. 2019] Stamoulis, D.; Ding, R.; Wang, D.; Lymberopoulos, D.; Priyantha, B.; Liu, J.; and Marculescu, D. 2019. Single-Path NAS: Designing HardwareEfficient ConvNets in less than 4 Hours. arXiv preprint. arXiv:1904.02877.
    Findings
  • [Tan et al. 2019] Tan, M.; Chen, B.; Pang, R.; Vasudevan, V.; and Le, Q. V. 2019. MnasNet: Platform-Aware Neural Architecture Search for Mobile. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.
    Google ScholarLocate open access versionFindings
  • [Wu et al. 2019] Wu, B.; Dai, X.; Zhang, P.; Wang, Y.; Sun, F.; Wu, Y.; Tian, Y.; Vajda, P.; Jia, Y.; and Keutzer, K. 2019. FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Search. In The IEEE Conference on Computer Vision and Pattern Recognition.
    Google ScholarLocate open access versionFindings
  • [Xiaomi 2018] Xiaomi. 2018. Mobile AI Compute Engine. https://github.com/XiaoMi/mace, commit hashtag:03362fa0.
    Findings
  • [Yang et al. 2018] Yang, T.-J.; Howard, A.; Chen, B.; Zhang, X.; Go, A.; Sandler, M.; Sze, V.; and Adam, H. 2018. NetAdapt: Platform-Aware Neural Network. Adaptation for
    Google ScholarFindings
  • Mobile Applications. In Proceedings of the European Conference on Computer Vision, 285–300.
    Google ScholarLocate open access versionFindings
  • [Zhang et al. 2018] Zhang, X.; Zhou, X.; Lin, M.; and Sun, J. 2018. ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices. In The IEEE Conference on Computer Vision and Pattern Recognition.
    Google ScholarLocate open access versionFindings
  • [Zoph et al. 2018] Zoph, B.; Vasudevan, V.; Shlens, J.; and Le, Q. V. 2018. Learning Transferable Architectures for Scalable Image Recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 8697–8710.
    Google ScholarLocate open access versionFindings
Your rating :
0

 

Tags
Comments