{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2024,9,20]],"date-time":"2024-09-20T16:52:43Z","timestamp":1726851163075},"reference-count":84,"publisher":"Association for Computing Machinery (ACM)","issue":"1","funder":[{"DOI":"10.13039\/501100001809","name":"Natural Science Foundation of China","doi-asserted-by":"crossref","award":["61732008 and 61902209"],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]},{"name":"Beijing Academy of Artificial Intelligence (BAAI) and Tsinghua University Guoqiang Research Institute, Beijing Outstanding Young Scientist Program","award":["BJJWZYJH012019100020098"]},{"name":"Intelligent Social Governance Platform, Major Innovation & Planning Interdisciplinary Platform for the \u201cDouble-First Class\u201d Initiative, Renmin University of China"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Inf. Syst."],"published-print":{"date-parts":[[2023,1,31]]},"abstract":"Result ranking is one of the major concerns for Web search technologies. Most existing methodologies rank search results in descending order of relevance. To model the interactions among search results, reinforcement learning (RL algorithms have been widely adopted for ranking tasks. However, the online training of RL methods is time and resource consuming at scale. As an alternative, learning ranking policies in the simulation environment is much more feasible and efficient. In this article, we propose two different simulation environments for the offline training of the RL ranking agent: the Context-aware Click Simulator (CCS) and the Fine-grained User Behavior Simulator with GAN (UserGAN). Based on the simulation environment, we also design a User Behavior Simulation for Reinforcement Learning (UBS4RL) re-ranking framework, which consists of three modules: a feature extractor for heterogeneous search results, a user simulator for collecting simulated user feedback, and a ranking agent for generation of optimized result lists. Extensive experiments on both simulated and practical Web search datasets show that (1) the proposed user simulators can capture and simulate fine-grained user behavior patterns by training on large-scale search logs, (2) the temporal information of user searching process is a strong signal for ranking evaluation, and (3) learning ranking policies from the simulation environment can effectively improve the search ranking performance.<\/jats:p>","DOI":"10.1145\/3511469","type":"journal-article","created":{"date-parts":[[2022,2,14]],"date-time":"2022-02-14T20:19:19Z","timestamp":1644869959000},"page":"1-35","update-policy":"http:\/\/dx.doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":3,"title":["User Behavior Simulation for Search Result Re-ranking"],"prefix":"10.1145","volume":"41","author":[{"ORCID":"http:\/\/orcid.org\/0000-0002-6234-8171","authenticated-orcid":false,"given":"Junqi","family":"Zhang","sequence":"first","affiliation":[{"name":"Tsinghua University, Beijing, China"}]},{"given":"Yiqun","family":"Liu","sequence":"additional","affiliation":[{"name":"Tsinghua University, Beijing, China"}]},{"given":"Jiaxin","family":"Mao","sequence":"additional","affiliation":[{"name":"Renmin University of China, Beijing, China"}]},{"given":"Weizhi","family":"Ma","sequence":"additional","affiliation":[{"name":"Tsinghua University, Beijing, China"}]},{"given":"Jiazheng","family":"Xu","sequence":"additional","affiliation":[{"name":"Tsinghua University, Beijing, China"}]},{"given":"Shaoping","family":"Ma","sequence":"additional","affiliation":[{"name":"Tsinghua University, Beijing, China"}]},{"given":"Qi","family":"Tian","sequence":"additional","affiliation":[{"name":"Huawei Cloud & AI, China"}]}],"member":"320","published-online":{"date-parts":[[2023,1,20]]},"reference":[{"key":"e_1_3_2_2_2","unstructured":"Keith Jack. 2008. Instant access. In Digital Video and DSP . Newnes Burlington MA 223\u2013231. 10.1016\/B978-0-7506-8975-5.00009-1"},{"key":"e_1_3_2_3_2","unstructured":"Qingyao Ai Keping Bi Luo Cheng Jiafeng Guo and W. Bruce Croft. 2018. Unbiased learning to rank with unbiased propensity estimation. In Proceedings of the 41st International ACM SIGIR Conference on Research and Development in Information Retrieval . 385\u2013394."},{"key":"e_1_3_2_4_2","unstructured":"Robert H. Berk. 1996. Continuous Univariate Distributions Vol. 2. Wiley Series in Probability and Statistics. Wiley."},{"key":"e_1_3_2_5_2","doi-asserted-by":"crossref","first-page":"205","DOI":"10.1145\/2911451.2911504","volume-title":"Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval","author":"Borisov Alexey","year":"2016","unstructured":"Alexey Borisov, Ilya Markov, Maarten de Rijke, and Pavel Serdyukov. 2016. A context-aware time model for web search. In Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, New York, NY, 205\u2013214."},{"key":"e_1_3_2_6_2","doi-asserted-by":"crossref","first-page":"205","DOI":"10.1145\/2911451.2911504","volume-title":"Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval","author":"Borisov Alexey","year":"2016","unstructured":"Alexey Borisov, Ilya Markov, Maarten de Rijke, and Pavel Serdyukov. 2016. A context-aware time model for web search. In Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval. 205\u2013214."},{"key":"e_1_3_2_7_2","volume-title":"Proceedings of the International Conference on World Wide Web","author":"Borisov Alexey","year":"2016","unstructured":"Alexey Borisov, Ilya Markov, Maarten De Rijke, and Pavel Serdyukov. 2016. A neural click model for web search. In Proceedings of the International Conference on World Wide Web."},{"key":"e_1_3_2_8_2","doi-asserted-by":"crossref","unstructured":"Alexey Borisov Martijn Wardenaar Ilya Markov and Maarten De Rijke. 2018. A click sequence model for web search. In Proceedings of the 41st International ACM SIGIR Conference on Research and Development in Information Retrieval . 45\u201354.","DOI":"10.1145\/3209978.3210004"},{"key":"e_1_3_2_9_2","volume-title":"Proceedings of the IEEE International Conference on Computer Vision","author":"Caicedo Juan C.","year":"2015","unstructured":"Juan C. Caicedo and Svetlana Lazebnik. 2015. Active object localization with deep reinforcement learning. In Proceedings of the IEEE International Conference on Computer Vision."},{"key":"e_1_3_2_10_2","volume-title":"The use of MMR, diversity-based reranking for reordering documents and producing summaries","author":"Carbonell Jaime","year":"1998","unstructured":"Jaime Carbonell and Jade Goldstein. 1998. The use of MMR, diversity-based reranking for reordering documents and producing summaries. In Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. 335\u2013336."},{"key":"e_1_3_2_11_2","volume-title":"Statistical Inference","author":"Casella George","year":"2002","unstructured":"George Casella and Roger L. Berger. 2002. Statistical Inference, Vol. 2. Duxbury, Pacific Grove, CA."},{"key":"e_1_3_2_12_2","doi-asserted-by":"crossref","first-page":"137","DOI":"10.1145\/3269206.3271743","volume-title":"Proceedings of the 27th ACM International Conference on Information and Knowledge Management","author":"Chae Dong-Kyu","year":"2018","unstructured":"Dong-Kyu Chae, Jin-Soo Kang, Sang-Wook Kim, and Jung-Tae Lee. 2018. CFGAN: A generic collaborative filtering framework based on generative adversarial networks. In Proceedings of the 27th ACM International Conference on Information and Knowledge Management. ACM, New York, NY, 137\u2013146."},{"key":"e_1_3_2_13_2","doi-asserted-by":"crossref","first-page":"621","DOI":"10.1145\/1645953.1646033","volume-title":"Proceedings of the 18th ACM Conference on Information and Knowledge Management","author":"Chapelle Olivier","year":"2009","unstructured":"Olivier Chapelle, Donald Metlzer, Ya Zhang, and Pierre Grinspan. 2009. Expected reciprocal rank for graded relevance. In Proceedings of the 18th ACM Conference on Information and Knowledge Management. ACM, New York, NY, 621\u2013630."},{"key":"e_1_3_2_14_2","first-page":"1","volume-title":"Proceedings of the 18th International Conference on World Wide Web","author":"Chapelle Olivier","year":"2009","unstructured":"Olivier Chapelle and Ya Zhang. 2009. A dynamic Bayesian network click model for web search ranking. In Proceedings of the 18th International Conference on World Wide Web. ACM, New York, NY, 1\u201310."},{"key":"e_1_3_2_15_2","article-title":"Maximum-likelihood augmented discrete generative adversarial networks","author":"Che Tong","year":"2017","unstructured":"Tong Che, Yanran Li, Ruixiang Zhang, R. Devon Hjelm, Wenjie Li, Yangqiu Song, and Yoshua Bengio. 2017. Maximum-likelihood augmented discrete generative adversarial networks. arXiv preprint arXiv:1702.07983 (2017).","journal-title":"arXiv preprint arXiv:1702.07983"},{"key":"e_1_3_2_16_2","doi-asserted-by":"crossref","first-page":"463","DOI":"10.1145\/2124295.2124351","volume-title":"Proceedings of the 5th ACM International Conference on Web Search and Data Mining","author":"Chen Danqi","year":"2012","unstructured":"Danqi Chen, Weizhu Chen, Haixun Wang, Zheng Chen, and Qiang Yang. 2012. Beyond ten blue links: Enabling user click modeling in federated web search. In Proceedings of the 5th ACM International Conference on Web Search and Data Mining. ACM, New York, NY, 463\u2013472."},{"key":"e_1_3_2_17_2","first-page":"815","volume-title":"Proceedings of the 41st International ACM SIGIR Conference on Research and Development in Information Retrieval","author":"Chen Qin","year":"2018","unstructured":"Qin Chen, Qinmin Hu, Jimmy Xiangji Huang, and Liang He. 2018. CAN: Enhancing sentence similarity modeling with collaborative and adversarial network. In Proceedings of the 41st International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, New York, NY, 815\u2013824."},{"key":"e_1_3_2_18_2","volume-title":"Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval","author":"Chen Ye","year":"2017","unstructured":"Ye Chen, Ke Zhou, Yiqun Liu, Min Zhang, and Shaoping Ma. 2017. Meta-evaluation of online and offline web search evaluation metrics. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, New York, NY, 15\u201324."},{"key":"e_1_3_2_19_2","doi-asserted-by":"crossref","first-page":"15","DOI":"10.1145\/3077136.3080804","volume-title":"Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval","author":"Chen Ye","year":"2017","unstructured":"Ye Chen, Ke Zhou, Yiqun Liu, Min Zhang, and Shaoping Ma. 2017. Meta-evaluation of online and offline web search evaluation metrics. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, New York, NY, 15\u201324."},{"key":"e_1_3_2_20_2","unstructured":"Kyunghyun Cho Bart Van Merrienboer Dzmitry Bahdanau and Yoshua Bengio. 2014. On the properties of neural machine translation: Encoder-decoder approaches. In Proceedings of the 8th Workshop on Syntax Semantics and Structure in Statistical Translation . 103\u2013111."},{"key":"e_1_3_2_21_2","doi-asserted-by":"crossref","first-page":"493","DOI":"10.1145\/2484028.2484071","volume-title":"Proceedings of the 36th International ACM SIGIR Conference on Research and Development in Information Retrieval","author":"Chuklin Aleksandr","year":"2013","unstructured":"Aleksandr Chuklin, Pavel Serdyukov, and Maarten De Rijke. 2013. Click model-based information retrieval metrics. In Proceedings of the 36th International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, New York, NY, 493\u2013502."},{"key":"e_1_3_2_22_2","volume-title":"Proceedings of the European Conference on Information Retrieval","author":"Chuklin Aleksandr","year":"2013","unstructured":"Aleksandr Chuklin, Pavel Serdyukov, and Maarten De Rijke. 2013. Using intent information to model user behavior in diversified search. In Proceedings of the European Conference on Information Retrieval."},{"key":"e_1_3_2_23_2","unstructured":"Cyril W. Cleverdon Jack Mills and E. Michael Keen. 1966. Factors Determining the Performance of Indexing Systems; Volume 1 Design; Part 1 Text . 1: Design. College of Aeronautics Cranfield."},{"issue":"4","key":"e_1_3_2_24_2","doi-asserted-by":"crossref","first-page":"213","DOI":"10.1037\/h0026256","article-title":"Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit.","volume":"70","author":"Cohen Jacob","year":"1968","unstructured":"Jacob Cohen. 1968. Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit.Psychological Bulletin 70, 4 (1968), 213.","journal-title":"Psychological Bulletin"},{"key":"e_1_3_2_25_2","doi-asserted-by":"crossref","first-page":"913","DOI":"10.1145\/3269206.3271768","volume-title":"Proceedings of the 27th ACM International Conference on Information and Knowledge Management","author":"Ding Ming","year":"2018","unstructured":"Ming Ding, Jie Tang, and Jie Zhang. 2018. Semi-supervised learning on graphs with generative adversarial nets. In Proceedings of the 27th ACM International Conference on Information and Knowledge Management. ACM, New York, NY, 913\u2013922."},{"key":"e_1_3_2_26_2","doi-asserted-by":"crossref","first-page":"331","DOI":"10.1145\/1390334.1390392","volume-title":"Proceedings of the 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval","author":"Dupret Georges E.","year":"2008","unstructured":"Georges E. Dupret and Benjamin Piwowarski. 2008. A user browsing model to predict search engine click data from past observations. In Proceedings of the 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, New York, NY, 331\u2013338."},{"key":"e_1_3_2_27_2","article-title":"MaskGAN: Better text generation via filling in the_","author":"Fedus William","year":"2018","unstructured":"William Fedus, Ian Goodfellow, and Andrew M. Dai. 2018. MaskGAN: Better text generation via filling in the_. arXiv preprint arXiv:1801.07736 (2018).","journal-title":"arXiv preprint arXiv:1801.07736"},{"key":"e_1_3_2_28_2","article-title":"NIPS 2016 tutorial: Generative adversarial networks","author":"Goodfellow Ian","year":"2016","unstructured":"Ian Goodfellow. 2016. NIPS 2016 tutorial: Generative adversarial networks. arXiv preprint arXiv:1701.00160 (2016).","journal-title":"arXiv preprint arXiv:1701.00160"},{"key":"e_1_3_2_29_2","first-page":"2672","volume-title":"Advances in Neural Information Processing Systems","author":"Goodfellow Ian","year":"2014","unstructured":"Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. In Advances in Neural Information Processing Systems. 2672\u20132680."},{"key":"e_1_3_2_30_2","first-page":"283","article-title":"Eyetracking in online search","author":"Granka Laura","year":"2008","unstructured":"Laura Granka, Matthew Feusner, and Lori Lorigo. 2008. Eyetracking in online search. In Passive Eye Monitoring. Springer, 283\u2013304.","journal-title":"Passive Eye Monitoring."},{"key":"e_1_3_2_31_2","doi-asserted-by":"crossref","first-page":"417","DOI":"10.1145\/1240624.1240691","volume-title":"Proceedings of the SIGCHI Conference on Human Factors in Computing Systems","author":"Guan Zhiwei","year":"2007","unstructured":"Zhiwei Guan and Edward Cutrell. 2007. An eye tracking study of the effect of target rank on web search. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 417\u2013420."},{"key":"e_1_3_2_32_2","article-title":"Objective-reinforced generative adversarial networks (ORGAN) for sequence generation models","author":"Guimaraes Gabriel Lima","year":"2017","unstructured":"Gabriel Lima Guimaraes, Benjamin Sanchez-Lengeling, Carlos Outeiral, Pedro Luis Cunha Farias, and Al\u00e1n Aspuru-Guzik. 2017. Objective-reinforced generative adversarial networks (ORGAN) for sequence generation models. arXiv preprint arXiv:1705.10843 (2017).","journal-title":"arXiv preprint arXiv:1705.10843"},{"key":"e_1_3_2_33_2","first-page":"124","volume-title":"Proceedings of the 2nd ACM International Conference on Web Search and Data Mining","author":"Guo Fan","year":"2009","unstructured":"Fan Guo, Chao Liu, and Yi Min Wang. 2009. Efficient multiple-click models in web search. In Proceedings of the 2nd ACM International Conference on Web Search and Data Mining. ACM, New York, NY, 124\u2013131."},{"key":"e_1_3_2_34_2","article-title":"World models","author":"Ha David","year":"2018","unstructured":"David Ha and J\u00fcrgen Schmidhuber. 2018. World models. arXiv preprint arXiv:1803.10122 (2018).","journal-title":"arXiv preprint arXiv:1803.10122"},{"key":"e_1_3_2_35_2","first-page":"2042","volume-title":"Advances in Neural Information Processing Systems","author":"Hu Baotian","year":"2014","unstructured":"Baotian Hu, Zhengdong Lu, Hang Li, and Qingcai Chen. 2014. Convolutional neural network architectures for matching natural language sentences. In Advances in Neural Information Processing Systems. 2042\u20132050."},{"key":"e_1_3_2_36_2","first-page":"784","volume-title":"Proceedings of the World Wide Web Conference","author":"Jia Yuting","year":"2019","unstructured":"Yuting Jia, Qinqin Zhang, Weinan Zhang, and Xinbing Wang. 2019. CommunityGAN: Community detection with generative adversarial nets. In Proceedings of the World Wide Web Conference. ACM, New York, NY, 784\u2013794."},{"key":"e_1_3_2_37_2","doi-asserted-by":"crossref","unstructured":"Thorsten Joachims Laura Granka Bing Pan Helene Hembrooke and Geri Gay. 2005. Accurately interpreting clickthrough data as implicit feedback. ACM SIGIR Forum 51 1 (2005) 4\u201311.","DOI":"10.1145\/3130332.3130334"},{"key":"e_1_3_2_38_2","doi-asserted-by":"crossref","first-page":"781","DOI":"10.1145\/3018661.3018699","volume-title":"Proceedings of the 10th ACM International Conference on Web Search and Data Mining","author":"Joachims Thorsten","year":"2017","unstructured":"Thorsten Joachims, Adith Swaminathan, and Tobias Schnabel. 2017. Unbiased learning-to-rank with biased feedback. In Proceedings of the 10th ACM International Conference on Web Search and Data Mining. 781\u2013789."},{"key":"e_1_3_2_39_2","unstructured":"Junbo Zhao Yoon Kim Kelly Zhang Alexander Rush and Yann LeCun. 2017. Adversarially regularized autoencoders for generating discrete structures. arXiv preprint arXiv:1706.04223 (2017)."},{"key":"e_1_3_2_40_2","doi-asserted-by":"crossref","unstructured":"Huang A. Maddison C. J. Guez A. D. Silver and D. Hassabis. 2016. Mastering the game of Go with deep neural 1139 networks and tree search. Nature 529 7587 484\u2013489.","DOI":"10.1038\/nature16961"},{"key":"e_1_3_2_41_2","article-title":"Auto-encoding variational Bayes","author":"Kingma Diederik P.","year":"2013","unstructured":"Diederik P. Kingma and Max Welling. 2013. Auto-encoding variational Bayes. arXiv preprint arXiv:1312.6114 (2013).","journal-title":"arXiv preprint arXiv:1312.6114"},{"key":"e_1_3_2_42_2","doi-asserted-by":"crossref","unstructured":"Levente Kocsis and Csaba Szepesvari. 2006. Bandit based Monte-Carlo planning. In Machine Learning: ECML 2006 . Lecture Notes in Computer Science Vol. 4212. Springer 282\u2013293.","DOI":"10.1007\/11871842_29"},{"key":"e_1_3_2_43_2","volume-title":"Technical Report DFVLR-FB 88-28. DLR German Aerospace Center\u2014Institute for Flight Mechanics, Koln, Germany.","author":"Kraft Dieter","year":"1988","unstructured":"Dieter Kraft. 1988. A Software Package for Sequential Quadratic Programming. In Technical Report DFVLR-FB 88-28. DLR German Aerospace Center\u2014Institute for Flight Mechanics, Koln, Germany."},{"key":"e_1_3_2_44_2","article-title":"Polyphonic music generation with sequence generative adversarial networks","author":"Lee Sang-Gil","year":"2017","unstructured":"Sang-Gil Lee, Uiwon Hwang, Seonwoo Min, and Sungroh Yoon. 2017. Polyphonic music generation with sequence generative adversarial networks. arXiv preprint arXiv:1710.11418 (2017).","journal-title":"arXiv preprint arXiv:1710.11418"},{"key":"e_1_3_2_45_2","doi-asserted-by":"crossref","first-page":"1039","DOI":"10.1145\/3308558.3313625","volume-title":"Proceedings of the World Wide Web Conference","author":"Liang Shangsong","year":"2019","unstructured":"Shangsong Liang. 2019. Unsupervised semantic generative adversarial networks for expert retrieval. In Proceedings of the World Wide Web Conference. ACM, New York, NY, 1039\u20131050."},{"key":"e_1_3_2_46_2","first-page":"3155","volume-title":"Advances in Neural Information Processing Systems","author":"Lin Kevin","year":"2017","unstructured":"Kevin Lin, Dianqi Li, Xiaodong He, Zhengyou Zhang, and Ming-Ting Sun. 2017. Adversarial ranking for language generation. In Advances in Neural Information Processing Systems. 3155\u20133165."},{"key":"e_1_3_2_47_2","first-page":"700","volume-title":"Advances in Neural Information Processing Systems","author":"Liu Ming-Yu","year":"2017","unstructured":"Ming-Yu Liu, Thomas Breuel, and Jan Kautz. 2017. Unsupervised image-to-image translation networks. In Advances in Neural Information Processing Systems. 700\u2013708."},{"issue":"3","key":"e_1_3_2_48_2","first-page":"1","article-title":"Time-aware click model","volume":"35","author":"Liu Yiqun","year":"2016","unstructured":"Yiqun Liu, Xiaohui Xie, Chao Wang, Jian-Yun Nie, Min Zhang, and Shaoping Ma. 2016. Time-aware click model. ACM Transactions on Information Systems 35, 3 (2016), 1\u201324.","journal-title":"ACM Transactions on Information Systems"},{"key":"e_1_3_2_49_2","doi-asserted-by":"publisher","DOI":"10.1145\/3329188"},{"key":"e_1_3_2_50_2","first-page":"555","volume-title":"Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval","author":"Lu Shuqi","year":"2019","unstructured":"Shuqi Lu, Zhicheng Dou, Xu Jun, Jian-Yun Nie, and Ji-Rong Wen. 2019. PSGAN: A minimax game for personalized search with limited and noisy click data. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, New York, NY, 555\u2013564."},{"key":"e_1_3_2_51_2","volume-title":"Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval","author":"Mao Jiaxin","year":"2018","unstructured":"Jiaxin Mao, Cheng Luo, Min Zhang, and Shaoping Ma. 2018. Constructing click models for mobile search. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, New York, NY."},{"key":"e_1_3_2_52_2","first-page":"2794","volume-title":"Proceedings of the IEEE International Conference on Computer Vision","author":"Mao Xudong","year":"2017","unstructured":"Xudong Mao, Qing Li, Haoran Xie, Raymond Y. K. Lau, Zhen Wang, and Stephen Paul Smolley. 2017. Least squares generative adversarial networks. In Proceedings of the IEEE International Conference on Computer Vision. 2794\u20132802."},{"key":"e_1_3_2_53_2","first-page":"731","volume-title":"Proceedings of the 25th ACM International Conference on Information and Knowledge Management","author":"Maxwell David","year":"2016","unstructured":"David Maxwell and Leif Azzopardi. 2016. Agents, simulated users and humans: An analysis of performance and behaviour. In Proceedings of the 25th ACM International Conference on Information and Knowledge Management. 731\u2013740."},{"key":"e_1_3_2_54_2","article-title":"Conditional generative adversarial nets","author":"Mirza Mehdi","year":"2014","unstructured":"Mehdi Mirza and Simon Osindero. 2014. Conditional generative adversarial nets. arXiv preprint arXiv:1411.1784 (2014).","journal-title":"arXiv preprint arXiv:1411.1784"},{"issue":"1","key":"e_1_3_2_55_2","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/1416950.1416952","article-title":"Rank-biased precision for measurement of retrieval","volume":"27","author":"Moffat Alistair","year":"2008","unstructured":"Alistair Moffat and Justin Zobel. 2008. Rank-biased precision for measurement of retrieval. ACM Transactions on Information Systems 27, 1 (2008), 1\u201327.","journal-title":"ACM Transactions on Information Systems"},{"key":"e_1_3_2_56_2","unstructured":"Seyed Sajad Mousavi Michael Schukat and Enda Howley. 2018. Deep reinforcement learning: An overview. arXiv preprint arXiv:1701-07274 (2018)."},{"key":"e_1_3_2_57_2","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Ng Andrew Y.","year":"2000","unstructured":"Andrew Y. Ng and Stuart Russell. 2000. Algorithms for inverse reinforcement learning. In Proceedings of the International Conference on Machine Learning."},{"key":"e_1_3_2_58_2","article-title":"WaveNet: A generative model for raw audio","author":"Oord Aaron van den","year":"2016","unstructured":"Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. 2016. WaveNet: A generative model for raw audio. arXiv preprint arXiv:1609.03499 (2016).","journal-title":"arXiv preprint arXiv:1609.03499"},{"key":"e_1_3_2_59_2","article-title":"Pixel recurrent neural networks","author":"Oord Aaron van den","year":"2016","unstructured":"Aaron van den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. 2016. Pixel recurrent neural networks. arXiv preprint arXiv:1601.06759 (2016).","journal-title":"arXiv preprint arXiv:1601.06759"},{"key":"e_1_3_2_60_2","doi-asserted-by":"crossref","first-page":"1293","DOI":"10.1145\/3269206.3271686","volume-title":"Proceedings of the 27th ACM International Conference on Information and Knowledge Management","author":"Oosterhuis Harrie","year":"2018","unstructured":"Harrie Oosterhuis and Maarten de Rijke. 2018. Differentiable unbiased online learning to rank. In Proceedings of the 27th ACM International Conference on Information and Knowledge Management. 1293\u20131302."},{"key":"e_1_3_2_61_2","first-page":"845","volume-title":"Proceedings of the 41st International ACM SIGIR Conference on Research and Development in Information Retrieval.","author":"Oosterhuis Harrie","year":"2018","unstructured":"Harrie Oosterhuis and Maarten De Rijke. 2018. Ranking for relevance and display preferences in complex presentation layouts. In Proceedings of the 41st International ACM SIGIR Conference on Research and Development in Information Retrieval.845\u2013854."},{"key":"e_1_3_2_62_2","doi-asserted-by":"crossref","first-page":"499","DOI":"10.1145\/3397271.3401104","volume-title":"Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval","author":"Pang Liang","year":"2020","unstructured":"Liang Pang, Jun Xu, Qingyao Ai, Yanyan Lan, Xueqi Cheng, and Jirong Wen. 2020. SetRank: Learning a permutation-invariant ranking model for information retrieval. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval. 499\u2013508."},{"issue":"4","key":"e_1_3_2_63_2","doi-asserted-by":"crossref","first-page":"346","DOI":"10.1007\/s10791-009-9123-y","article-title":"LETOR: A benchmark collection for research on learning to rank for information retrieval.","volume":"13","author":"Qin Tao","year":"2010","unstructured":"Tao Qin, Tie Yan Liu, Jun Xu, and Hang Li. 2010. LETOR: A benchmark collection for research on learning to rank for information retrieval.Information Retrieval 13, 4 (2010), 346\u2013374.","journal-title":"Information Retrieval"},{"key":"e_1_3_2_64_2","article-title":"Unsupervised representation learning with deep convolutional generative adversarial networks","author":"Radford Alec","year":"2015","unstructured":"Alec Radford, Luke Metz, and Soumith Chintala. 2015. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434 (2015).","journal-title":"arXiv preprint arXiv:1511.06434"},{"issue":"4","key":"e_1_3_2_65_2","doi-asserted-by":"crossref","first-page":"422","DOI":"10.1145\/582415.582418","article-title":"Cumulated gain-based evaluation of IR techniques","volume":"20","author":"Rvelin Kalervo","year":"2002","unstructured":"Kalervo Rvelin, Kek, and Jaana Inen. 2002. Cumulated gain-based evaluation of IR techniques. ACM Transactions on Information Systems 20, 4 (2002), 422\u2013446.","journal-title":"ACM Transactions on Information Systems"},{"key":"e_1_3_2_66_2","doi-asserted-by":"crossref","first-page":"457","DOI":"10.1145\/2835776.2835804","volume-title":"Proceedings of the 9th ACM International Conference on Web Search and Data Mining","author":"Schuth Anne","year":"2016","unstructured":"Anne Schuth, Harrie Oosterhuis, Shimon Whiteson, and Maarten de Rijke. 2016. Multileave gradient descent for fast online learning to rank. In Proceedings of the 9th ACM International Conference on Web Search and Data Mining. 457\u2013466."},{"key":"e_1_3_2_67_2","first-page":"33","volume-title":"Proceedings of the European Conference on Computer Vision","author":"Shen Chengyao","year":"2014","unstructured":"Chengyao Shen and Qi Zhao. 2014. Webpage saliency. In Proceedings of the European Conference on Computer Vision. 33\u201346."},{"key":"e_1_3_2_68_2","unstructured":"Jing-Cheng Shi Yang Yu Qing Da Shi-Yong Chen and An-Xiang Zeng. 2018. Virtual-Taobao: Virtualizing real-world online retail environment for reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence . 845\u2013854."},{"key":"e_1_3_2_69_2","unstructured":"Richard S. Sutton and Andrew G. Barto. 2011. Reinforcement learning: An introduction. MIT Press."},{"key":"e_1_3_2_70_2","unstructured":"Pei Hao Su Milica Gasic Nikola Mrksic Lina M. Rojas Barahona Stefan Ultes David Vandyke Tsung Hsien Wen and Steve Young. 2016. On-line active reward learning for policy optimisation in spoken dialogue systems. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) . 2431\u20132441."},{"issue":"7540","key":"e_1_3_2_71_2","doi-asserted-by":"crossref","first-page":"529","DOI":"10.1038\/nature14236","article-title":"Human-level control through deep reinforcement learning","volume":"518","author":"Volodymyr Mnih","year":"2015","unstructured":"Mnih Volodymyr, Kavukcuoglu Koray, Silver David, Andrei A. Rusu, Veness Joel, Marc G. Bellemare, Graves Alex, Riedmiller Martin, Andreas K. Fidjeland, and Ostrovski Georg. 2015. Human-level control through deep reinforcement learning. Nature 518, 7540 (2015), 529.","journal-title":"Nature"},{"key":"e_1_3_2_72_2","doi-asserted-by":"crossref","first-page":"503","DOI":"10.1145\/2484028.2484036","volume-title":"Proceedings of the 36th international ACM SIGIR Conference on Research and Development in Information Retrieval","author":"Wang Chao","year":"2013","unstructured":"Chao Wang, Yiqun Liu, Min Zhang, Shaoping Ma, Meihong Zheng, Jing Qian, and Kuo Zhang. 2013. Incorporating vertical results into search click models. In Proceedings of the 36th international ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, New York, NY, 503\u2013512."},{"key":"e_1_3_2_73_2","doi-asserted-by":"crossref","first-page":"515","DOI":"10.1145\/3077136.3080786","volume-title":"Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval","author":"Wang Jun","year":"2017","unstructured":"Jun Wang, Lantao Yu, Weinan Zhang, Yu Gong, Yinghui Xu, Benyou Wang, Peng Zhang, and Dell Zhang. 2017. IRGAN: A minimax game for unifying generative and discriminative information retrieval models. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, New York, NY, 515\u2013524."},{"key":"e_1_3_2_74_2","doi-asserted-by":"crossref","first-page":"115","DOI":"10.1145\/2911451.2911537","volume-title":"Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval","author":"Wang Xuanhui","year":"2016","unstructured":"Xuanhui Wang, Michael Bendersky, Donald Metzler, and Marc Najork. 2016. Learning to rank with selection bias in personal search. In Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval. 115\u2013124."},{"key":"e_1_3_2_75_2","first-page":"610","volume-title":"Proceedings of the 11th ACM International Conference on Web Search and Data Mining","author":"Wang Xuanhui","year":"2018","unstructured":"Xuanhui Wang, Nadav Golbandi, Michael Bendersky, Donald Metzler, and Marc Najork. 2018. Position bias estimation for unbiased learning to rank in personal search. In Proceedings of the 11th ACM International Conference on Web Search and Data Mining. 610\u2013618."},{"issue":"3","key":"e_1_3_2_76_2","first-page":"19","article-title":"Optimizing whole-page presentation for web search","volume":"12","author":"Wang Yue","year":"2018","unstructured":"Yue Wang, Dawei Yin, Luo Jie, Pengyuan Wang, Makoto Yamada, Yi Chang, and Qiaozhu Mei. 2018. Optimizing whole-page presentation for web search. ACM Transactions on the Web 12, 3 (2018), 19.","journal-title":"ACM Transactions on the Web"},{"key":"e_1_3_2_77_2","volume-title":"Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval","author":"Wei Zeng","year":"2017","unstructured":"Zeng Wei, Jun Xu, Yanyan Lan, Jiafeng Guo, and Xueqi Cheng. 2017. Reinforcement learning to rank with Markov decision process. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval. 945\u2013948."},{"key":"e_1_3_2_78_2","volume-title":"Proceedings of the International ACM SIGIR Conference on Research and Development in Information Retrieval","author":"Xia Long","year":"2017","unstructured":"Long Xia, Jun Xu, Yanyan Lan, Jiafeng Guo, and Xueqi Cheng. 2017. Adapting Markov decision process for search result diversification. In Proceedings of the International ACM SIGIR Conference on Research and Development in Information Retrieval."},{"key":"e_1_3_2_79_2","volume-title":"Proceedings of the 31st AAAI Conference on Artificial Intelligence","author":"Yu Lantao","year":"2017","unstructured":"Lantao Yu, Weinan Zhang, Jun Wang, and Yong Yu. 2017. SeqGAN: Sequence generative adversarial nets with policy gradient. In Proceedings of the 31st AAAI Conference on Artificial Intelligence."},{"key":"e_1_3_2_80_2","article-title":"Towards vision-based deep reinforcement learning for robotic motion control","author":"Zhang Fangyi","year":"2015","unstructured":"Fangyi Zhang, Juergen Leitner, Michael Milford, Ben Upcroft, and Peter Corke. 2015. Towards vision-based deep reinforcement learning for robotic motion control. arXiv preprint arXiv:1511.03791 (2015).","journal-title":"arXiv preprint arXiv:1511.03791"},{"key":"e_1_3_2_81_2","volume-title":"Proceedings of the 2018 ACM on Conference on Information and Knowledge Management","author":"Zhang Junqi","year":"2018","unstructured":"Junqi Zhang, Yiqun Liu, Shaoping Ma, and Qi Tian. 2018. Relevance estimation with multiple information sources on search engine result pages. In Proceedings of the 2018 ACM on Conference on Information and Knowledge Management. ACM, New York, NY, 1\u201310."},{"key":"e_1_3_2_82_2","doi-asserted-by":"crossref","first-page":"1603","DOI":"10.1145\/3357384.3357945","volume-title":"Proceedings of the 28th ACM International Conference on Information and Knowledge Management","author":"Zhang Junqi","year":"2019","unstructured":"Junqi Zhang, Jiaxin Mao, Yiqun Liu, Ruizhe Zhang, Min Zhang, Shaoping Ma, Jun Xu, and Qi Tian. 2019. Context-aware ranking by constructing a virtual environment for reinforcement learning. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management. ACM, New York, NY, 1603\u20131612."},{"key":"e_1_3_2_83_2","first-page":"4006","volume-title":"Proceedings of the 34th International Conference on Machine Learning\u2014Volume 70","author":"Zhang Yizhe","year":"2017","unstructured":"Yizhe Zhang, Zhe Gan, Kai Fan, Zhi Chen, Ricardo Henao, Dinghan Shen, and Lawrence Carin. 2017. Adversarial feature matching for text generation. In Proceedings of the 34th International Conference on Machine Learning\u2014Volume 70. 4006\u20134015."},{"key":"e_1_3_2_84_2","first-page":"2223","volume-title":"Proceedings of the IEEE International Conference on Computer Vision","author":"Zhu Jun-Yan","year":"2017","unstructured":"Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A. Efros. 2017. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE International Conference on Computer Vision. 2223\u20132232."},{"key":"e_1_3_2_85_2","volume-title":"Proceedings of the International ACM SIGIR Conference on Research and Development in Information Retrieval","author":"Zhu Yadong","year":"2014","unstructured":"Yadong Zhu, Yanyan Lan, Jiafeng Guo, Xueqi Cheng, and Shuzi Niu. 2014. Learning for search result diversification. In Proceedings of the International ACM SIGIR Conference on Research and Development in Information Retrieval."}],"container-title":["ACM Transactions on Information Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3511469","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,1,20]],"date-time":"2023-01-20T10:11:59Z","timestamp":1674209519000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3511469"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,1,20]]},"references-count":84,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2023,1,31]]}},"alternative-id":["10.1145\/3511469"],"URL":"https:\/\/doi.org\/10.1145\/3511469","relation":{},"ISSN":["1046-8188","1558-2868"],"issn-type":[{"value":"1046-8188","type":"print"},{"value":"1558-2868","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,1,20]]},"assertion":[{"value":"2020-12-16","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2022-01-12","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-01-20","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}
