Abstract
Automatic text generation is widely used in dialogue systems, machine translation and other fields. Sequence Generative Adversarial Network (SeqGAN) has achieved good performance in text generation tasks. Due to the discriminator can only evaluate the finished text, and cannot provide other valid information to the generator. When evaluating a single word, the Monte Carlo algorithm is mainly used to generate a complete text. This process requires a huge computational cost. As the length of the text increases, the time to obtain rewards will increase significantly. For text, different words have different effects on semantics, and keywords determine the final expression of semantics. Evaluation of the importance of each word is particularly critical. In this paper, we propose a new framework called AttGAN. We allow the discriminator to provide more features to the generator. Specifically, we add an attention layer to the new discriminator. The attention score is used as the basic reward so that the discriminator can calculate the reward value of each word through only one evaluation without multiple sampling by the generator. And to meet the requirements of valid reward, we further process the attention score. Our large number of experiments on synthetic data and tests on dialogue systems show that AttGAN can minimize computational costs and generate high-quality text. Furthermore, it also has a good performance in the generation of lengthy text.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Bahdanau, D., Cho, K., Bengio, Y.: Neural machine translation by jointly learning to align and translate. In: 3rd International Conference on Learning Representations, San Diego (2015)
Bengio, S., Vinyals, O., Jaitly, N., Shazeer, N.: Scheduled sampling for sequence prediction with recurrent neural networks. In: Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, pp. 1171–1179. Neural Information Processing Systems, Montreal, Quebec, Canada (2015)
Cho, K., et al.: Learning phrase representations using RNN encoder-decoder for statistical machine translation. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, pp. 1724–1734. Association for Computational Linguistics, Doha (2014)
Elman, J.L.: Finding structure in time. Cogn. Sci. 14(2), 179–211 (1990)
Goodfellow, I.J., et al.: Generative adversarial nets. In: Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014, pp. 2672–2680. Neural Information Processing Systems, Montreal (2014)
Guo, J., Lu, S., Cai, H., Zhang, W., Yu, Y., Wang, J.: Long text generation via adversarial training with leaked information. In: Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, pp. 5141–5148. AAAI Press, New Orleans (2018)
Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997)
Kim, Y.: Convolutional neural networks for sentence classification. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, pp. 1746–1751. Association for Computational Linguistics, Doha (2014)
Li, J., Monroe, W., Shi, T., Jean, S., Ritter, A., Jurafsky, D.: Adversarial learning for neural dialogue generation. In: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 2157–2169. Association for Computational Linguistics, Copenhagen (2017)
Li, Y., Su, H., Shen, X., Li, W., Cao, Z., Niu, S.: DailyDialog: a manually labelled multi-turn dialogue dataset. In: Proceedings of the Eighth International Joint Conference on Natural Language Processing, pp. 986–995. Asian Federation of Natural Language Processing, Taipei (2017)
Lin, K., Li, D., He, X., Sun, M., Zhang, Z.: Adversarial ranking for language generation. In: Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, pp. 3155–3165. Neural Information Processing Systems, CA, USA (2017)
Salakhutdinov, R.: Learning deep generative models. Annu. Rev. Stat. Appl. 2, 361–385 (2015)
Song, H., Zhang, W., Hu, J., Liu, T.: Generating persona consistent dialogues by exploiting natural language inference. In: The Thirty-Fourth AAAI Conference on Artificial Intelligence, pp. 8878–8885. AAAI Press, New York (2020)
Srivastava, N., Hinton, G.E., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.: Dropout: a simple way to prevent neural networks from overfitting. J. Mach. Learn. Res. 15(1), 1929–1958 (2014)
Sutskever, I., Vinyals, O., Le, Q.V.: Sequence to sequence learning with neural networks. In: Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014, pp. 3104–3112. Neural Information Processing Systems, Montreal (2014)
Sutton, R.S., McAllester, D.A., Singh, S.P., Mansour, Y.: Policy gradient methods for reinforcement learning with function approximation. Advances in Neural Information Processing Systems, vol. 12, pp. 1057–1063. The MIT Press, Denver (1999)
Yang, Z., Chen, W., Wang, F., Xu, B.: Improving neural machine translation with conditional sequence generative adversarial nets. In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 1346–1355. Association for Computational Linguistics, New Orleans (2018)
Yang, Z., Yang, D., Dyer, C., He, X., Smola, A.J., Hovy, E.H.: Hierarchical attention networks for document classification. In: The 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 1480–1489. Association for Computational Linguistics, San Diego (2016)
Yu, L., Zhang, W., Wang, J., Yu, Y.: SeqGAN: sequence generative adversarial nets with policy gradient. In: Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, pp. 2852–2858. AAAI Press, San Francisco (2017)
Zhang, Y., et al.: Adversarial feature matching for text generation. In: Proceedings of the 34th International Conference on Machine Learning, pp. 4006–4015. PMLR, Sydney (2017)
Zhou, P., et al.: Attention-based bidirectional long short-term memory networks for relation classification. In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pp. 207–212. Association for Computer Linguistics, Berlin (2016)
Acknowledgment
This work is supported by the National Natural Science Foundation of China (NSFC) (61972455), Key Research and Development Program of Hubei Province (No. 2020BAB026) and the Joint Project of Bayescom. Xiaowang Zhang is supported by the program of Peiyang Young Scholars in Tianjin University (2019XRX-0032).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Gao, M., Zhang, S., Zhang, X., Feng, Z. (2021). An Attention-Based Approach to Accelerating Sequence Generative Adversarial Nets. In: Pham, D.N., Theeramunkong, T., Governatori, G., Liu, F. (eds) PRICAI 2021: Trends in Artificial Intelligence. PRICAI 2021. Lecture Notes in Computer Science(), vol 13032. Springer, Cham. https://doi.org/10.1007/978-3-030-89363-7_3
Download citation
DOI: https://doi.org/10.1007/978-3-030-89363-7_3
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-89362-0
Online ISBN: 978-3-030-89363-7
eBook Packages: Computer ScienceComputer Science (R0)