Abstract
The model size and computation requirement of Deep Convolutional Neural Networks (DNNs) have ever increased as their applications to various real-life use-cases, e.g., autonomous driving, are getting more pervasive and popular. While DNN workloads are executed on Graphics Processing Units (GPUs) in many cases, it is not trivial to improve the inference speed through the conventional DNN weight pruning techniques, due to the parallel architecture of GPUs. On the other hand, the coarse-grain pruning, also known as structured sparsity or structured pruning, can speedup the inference, but cause significant losses of accuracy. In this paper, we propose two fine-grain DNN pruning techniques that are aware of the underlying GPU architecture. For that, we analyze the hierarchical architecture of parallel processing elements and memory of GPU to identify the finest possible pruning where the removed weights can be safely skipped during the inference. The effectiveness of the proposed techniques has been evaluated with VGG16. Compared to the existing pruning techniques, the proposed methods result in significantly improved inference speed with less accuracy drop.
This work was supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. 2018-0-00769, Neuromorphic Computing Software Platform for Artificial Intelligence Systems).
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Bottleson, J., Kim, S., Andrews, J., Bindu, P., Murthy, D.N., Jin, J.: clCaffe: OpenCL accelerated Caffe for convolutional neural networks. In: 2016 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW), pp. 50–57. IEEE (2016)
Chellapilla, K., Puri, S., Simard, P.: High performance convolutional neural networks for document processing. In: Tenth International Workshop on Frontiers in Handwriting Recognition. Suvisoft (2006)
Chen, X.: Escort: efficient sparse convolutional neural networks on GPUs. arXiv preprint arXiv:1802.10280 (2018)
Cheng, Y., Wang, D., Zhou, P., Zhang, T.: A survey of model compression and acceleration for deep neural networks. arXiv preprint arXiv:1710.09282 (2017)
Greathouse, J.L., Knox, K., Poła, J., Varaganti, K., Daga, M.: clSPARSE: a vendor-optimized open-source sparse BLAS library. In: Proceedings of the 4th International Workshop on OpenCL, pp. 1–4 (2016)
Gu, J., Liu, Y., Gao, Y., Zhu, M.: OpenCL caffe: accelerating and enabling a cross platform machine learning framework. In: Proceedings of the 4th International Workshop on OpenCL, pp. 1–5 (2016)
Han, S., et al.: EIE: efficient inference engine on compressed deep neural network. In: 2016 ACM/IEEE 43rd Annual International Symposium on Computer Architecture (ISCA), pp. 243–254. IEEE (2016)
Han, S., Pool, J., Tran, J., Dally, W.: Learning both weights and connections for efficient neural network. In: Advances in Neural Information Processing Systems, pp. 1135–1143 (2015)
Krizhevsky, A., Hinton, G., et al.: Learning multiple layers of features from tiny images (2009)
Krizhevsky, A., Sutskever, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems, pp. 1097–1105 (2012)
Li, H., Kadav, A., Durdanovic, I., Samet, H., Graf, H.P.: Pruning filters for efficient convnets. arXiv preprint arXiv:1608.08710 (2016)
Naumov, M., Chien, L., Vandermersch, P., Kapasi, U.: cuSPARSE library. In: GPU Technology Conference (2010)
Paszke, A., et al.: PyTorch: an imperative style, high-performance deep learning library. In: Wallach, H., Larochelle, H., Beygelzimer, A., d’Alché-Buc, F., Fox, E., Garnett, R. (eds.) Advances in Neural Information Processing Systems 32, pp. 8024–8035. Curran Associates, Inc. (2019). http://papers.neurips.cc/paper/9015-pytorch-an-imperative-style-high-performance-deep-learning-library.pdf
Perkins, H.: cltorch: a hardware-agnostic backend for the torch deep neural network library, based on OpenCL. arXiv preprint arXiv:1606.04884 (2016)
Radu, V., et al.: Performance aware convolutional neural network channel pruning for embedded GPUs. In: 2019 IEEE International Symposium on Workload Characterization (IISWC), pp. 24–34. IEEE (2019)
Redmon, J.: Darknet: Open source neural networks in C (2013)
Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv e-prints arXiv:1409.1556, September 2014
Stone, J.E., Gohara, D., Shi, G.: OpenCL: a parallel programming standard for heterogeneous computing systems. Comput. Sci. Eng. 12(3), 66 (2010)
Tanner, D.: Autogemm, October 2015. https://github.com/clMathLibraries/clBLAS/wiki/AutoGemm
Wen, W., Wu, C., Wang, Y., Chen, Y., Li, H.: Learning structured sparsity in deep neural networks. In: Advances in Neural Information Processing Systems, pp. 2074–2082 (2016)
Yao, Z., Cao, S., Xiao, W., Zhang, C., Nie, L.: Balanced sparsity for efficient DNN inference on GPU. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 5676–5683 (2019)
Zhu, J., Jiang, J., Chen, X., Tsui, C.Y.: SparseNN: an energy-efficient neural network accelerator exploiting input and output sparsity. In: 2018 Design, Automation & Test in Europe Conference & Exhibition (DATE), pp. 241–244. IEEE (2018)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Choi, K., Yang, H. (2021). A GPU Architecture Aware Fine-Grain Pruning Technique for Deep Neural Networks. In: Sousa, L., Roma, N., Tomás, P. (eds) Euro-Par 2021: Parallel Processing. Euro-Par 2021. Lecture Notes in Computer Science(), vol 12820. Springer, Cham. https://doi.org/10.1007/978-3-030-85665-6_14
Download citation
DOI: https://doi.org/10.1007/978-3-030-85665-6_14
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-85664-9
Online ISBN: 978-3-030-85665-6
eBook Packages: Computer ScienceComputer Science (R0)