Abstract
This work proposes a novel Graph-based neural ArchiTecture Encoding Scheme, a.k.a. GATES, to improve the predictor-based neural architecture search. Specifically, different from existing graph-based schemes, GATES models the operations as the transformation of the propagating information, which mimics the actual data processing of neural architecture. GATES is a more reasonable modeling of the neural architectures, and can encode architectures from both the “operation on node” and “operation on edge” cell search spaces consistently. Experimental results on various search spaces confirm GATES’s effectiveness in improving the performance predictor. Furthermore, equipped with the improved performance predictor, the sample efficiency of the predictor-based neural architecture search (NAS) flow is boosted.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
Figure 2 illustrates the OON and OOE search spaces.
- 2.
A more comprehensive comparison of the MSE regression loss and multiple ranking losses is shown in the appendix.
- 3.
Note that this inner search component could be easily substituted with other search strategies.
- 4.
See “Setup and Additional Results” section in the appendix for more details.
- 5.
We also implement an ad-hoc solution of applying GCN on OOE architectures referred to as the Line Graph GCN solution, in which the graph is first converted to a line graph. See “Setup and Additional Results” section in the appendix for more details.
References
Chen, W., Liu, T.-Y., Lan, Y., Ma, Z., Li, H.: Ranking measures and loss functions in learning to rank. In: Bengio, Y., Schuurmans, D., Lafferty, J.D., Williams, C.K.I., Culotta, A. (eds.) Advances in Neural Information Processing Systems 22, pp. 315–323. Curran Associates, Inc., New York (2009)
Dong, X., Yang, Y.: Nas-bench-201: extending the scope of reproducible neural architecture search. In: International Conference on Learning Representations (2020). https://openreview.net/forum?id=HJxyZkBKDr
Guo, Y., et al.: Breaking the curse of space explosion: towards effcient nas with curriculum search. In: International Conference on Machine Learning (2010)
Guo, Y., et al.: Nat: neural architecture transformer for accurate and compact architectures. In: Advances in Neural Information Processing Systems, pp. 735–747 (2019)
Kandasamy, K., Neiswanger, W., Schneider, J., Poczos, B., Xing, E.P.: Neural architecture search with Bayesian optimisation and optimal transport. In: Advances in Neural Information Processing Systems, pp. 2016–2025 (2018)
Kipf, T.N., Welling, M.: Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907 (2016)
Lian, D., et al.: Towards fast adaptation of neural architectures with meta learning. In: International Conference on Learning Representations (2020)
Liu, C., et al.: Progressive neural architecture search. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11205, pp. 19–35. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01246-5_2
Liu, H., Simonyan, K., Vinyals, O., Fernando, C., Kavukcuoglu, K.: Hierarchical representations for efficient architecture search. arXiv preprint arXiv:1711.00436 (2017)
Liu, H., Simonyan, K., Yang, Y.: Darts: Differentiable architecture search. arXiv preprint arXiv:1806.09055 (2018)
Liu, T.Y., et al.: Learning to rank for information retrieval. Found. Trends® Inf. Retrieval 3(3), 225–331 (2009)
Luo, R., Qin, T., Chen, E.: Understanding and improving one-shot neural architecture optimization. arXiv preprint arXiv:1909.10815 (2019)
Luo, R., Tian, F., Qin, T., Chen, E., Liu, T.Y.: Neural architecture optimization. In: Bengio, S., Wallach, H., Larochelle, H., Grauman, K., Cesa-Bianchi, N., Garnett, R. (eds.) Advances in Neural Information Processing Systems 31, pp. 7816–7827. Curran Associates, Inc. (2018). http://papers.nips.cc/paper/8007-neural-architecture-optimization.pdf
Negrinho, R., Gordon, G.: Deeparchitect: automatically designing and training deep architectures. arXiv preprint arXiv:1704.08792 (2017)
Pham, H., Guan, M.Y., Zoph, B., Le, Q.V., Dean, J.: Efficient neural architecture search via parameter sharing. arXiv preprint arXiv:1802.03268 (2018)
Real, E., Aggarwal, A., Huang, Y., Le, Q.V.: Regularized evolution for image classifier architecture search. Proc. AAAI Conf. Artificial Intell. 33, 4780–4789 (2019)
Sciuto, C., Yu, K., Jaggi, M., Musat, C., Salzmann, M.: Evaluating the search phase of neural architecture search. arXiv preprint arXiv:1902.08142 (2019)
Sen, P.K.: Estimates of the regression coefficient based on Kendall’s tau. J. Am. Stat. Assoc. 63(324), 1379–1389 (1968)
Shi, H., Pi, R., Xu, H., Li, Z., Kwok, J.T., Zhang, T.: Multi-objective neural architecture search via predictive network performance optimization. arXiv preprint arXiv:1911.09336 (2019)
Stagge, P., Igel, C.: Neural network structures and isomorphisms: random walk characteristics of the search space. In: 2000 IEEE Symposium on Combinations of Evolutionary Computation and Neural Networks. Proceedings of the First IEEE Symposium on Combinations of Evolutionary Computation and Neural Networks (Cat. No. 00), pp. 82–90. IEEE (2000)
Wang, L., Zhao, Y., Jinnai, Y., Fonseca, R.: Alphax: exploring neural architectures with deep neural networks and monte carlo tree search. arXiv preprint arXiv:1805.07440 (2018)
Xie, S., Kirillov, A., Girshick, R., He, K.: Exploring randomly wired neural networks for image recognition. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1284–1293 (2019)
Xu, Y., et al.: Renas:relativistic evaluation of neural architecture search (2019)
Ying, C., Klein, A., Real, E., Christiansen, E., Murphy, K., Hutter, F.: Nas-bench-101: towards reproducible neural architecture search. arXiv preprint arXiv:1902.09635 (2019)
Zoph, B., Le, Q.V.: Neural architecture search with reinforcement learning. In: ICLR (2017). https://arxiv.org/abs/1611.01578
Acknowledgments
This work was supported by National Natural Science Foundation of China (No. 61832007, 61622403, 61621091, U19B2019), Beijing National Research Center for Information Science and Technology (BNRist). The authors thank Novauto for the support.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Ning, X., Zheng, Y., Zhao, T., Wang, Y., Yang, H. (2020). A Generic Graph-Based Neural Architecture Encoding Scheme for Predictor-Based NAS. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, JM. (eds) Computer Vision – ECCV 2020. ECCV 2020. Lecture Notes in Computer Science(), vol 12358. Springer, Cham. https://doi.org/10.1007/978-3-030-58601-0_12
Download citation
DOI: https://doi.org/10.1007/978-3-030-58601-0_12
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-58600-3
Online ISBN: 978-3-030-58601-0
eBook Packages: Computer ScienceComputer Science (R0)