{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2024,8,8]],"date-time":"2024-08-08T00:12:37Z","timestamp":1723075957775},"reference-count":69,"publisher":"Association for Computing Machinery (ACM)","issue":"4","funder":[{"name":"HK GRF","award":["17208223"]},{"name":"HK RIF","award":["R7030-22"]},{"name":"HKU-SCF FinTech Academy R\\&D Funding Scheme","award":["2021 and 2022"]},{"name":"HK ITF","award":["GHP\/169\/20SZ"]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Proc. ACM Manag. Data"],"published-print":{"date-parts":[[2023,12,8]]},"abstract":"The advent of data-intensive applications has fueled the evolution of hybrid transactional and analytical processing (HTAP). To support mixed workloads, distributed HTAP databases typically maintain two data copies that are specially tailored for data freshness and performance isolation. In particular, a copy in a row-oriented format is well-suited for OLTP workloads, and a second copy in a column-oriented format is optimized for OLAP workloads. Such a hybrid design opens up a new design space for query optimization: plans can be optimized over different data formats and can be executed over isolated resources, which we term hybrid plans. In this paper, we demonstrate that hybrid plans can largely benefit query execution (e.g., up to 11x speedups in our evaluation). However, we also found these benefits will potentially be at the cost of sacrificing data freshness or performance isolation since traditional optimizers may not precisely model and schedule the execution of hybrid plans on real-time updated HTAP databases. Therefore, we propose Metis, an HTAP-aware optimizer. We show, both theoretically and experimentally, that using the proposed optimizations, a system can largely benefit from hybrid plans while preserving isolated performance for OLTP and OLAP, and these optimizations are robust to the changes in workloads.<\/jats:p>","DOI":"10.1145\/3626750","type":"journal-article","created":{"date-parts":[[2023,12,12]],"date-time":"2023-12-12T19:01:21Z","timestamp":1702407681000},"page":"1-27","source":"Crossref","is-referenced-by-count":0,"title":["Rethink Query Optimization in HTAP Databases"],"prefix":"10.1145","volume":"1","author":[{"ORCID":"http:\/\/orcid.org\/0009-0000-5952-5168","authenticated-orcid":false,"given":"Haoze","family":"Song","sequence":"first","affiliation":[{"name":"The University of Hong Kong, Hong Kong, Hong Kong"}]},{"ORCID":"http:\/\/orcid.org\/0009-0002-2689-6020","authenticated-orcid":false,"given":"Wenchao","family":"Zhou","sequence":"additional","affiliation":[{"name":"Alibaba Group, Hangzhou, China"}]},{"ORCID":"http:\/\/orcid.org\/0009-0003-0770-5775","authenticated-orcid":false,"given":"Feifei","family":"Li","sequence":"additional","affiliation":[{"name":"Alibaba Group, Hangzhou, China"}]},{"ORCID":"http:\/\/orcid.org\/0009-0008-6355-4525","authenticated-orcid":false,"given":"Xiang","family":"Peng","sequence":"additional","affiliation":[{"name":"Alibaba Group, Hangzhou, China"}]},{"ORCID":"http:\/\/orcid.org\/0000-0001-7746-440X","authenticated-orcid":false,"given":"Heming","family":"Cui","sequence":"additional","affiliation":[{"name":"The University of Hong Kong, Hong Kong, Hong Kong"}]}],"member":"320","published-online":{"date-parts":[[2023,12,12]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/1376616.1376712"},{"volume-title":"Proteus: Autonomous Adaptive Storage for Mixed Workloads. Technical Report. Technical Report","year":"2022","author":"Abebe Michael","key":"e_1_2_1_2_1","unstructured":"Michael Abebe, Horatiu Lazu, and Khuzaima Daudjee. 2022. Proteus: Autonomous Adaptive Storage for Mixed Workloads. Technical Report. Technical Report. University of Waterloo. https:\/\/cs. uwaterloo. ca . . . ."},{"key":"e_1_2_1_3_1","first-page":"281","article-title":"Reoptimizing Data Parallel Computing","volume":"12","author":"Agarwal Sameer","year":"2012","unstructured":"Sameer Agarwal, Srikanth Kandula, Nicolas Bruno, Ming-ChuanWu, Ion Stoica, and Jingren Zhou. 2012. Reoptimizing Data Parallel Computing.. In NSDI, Vol. 12. 281--294.","journal-title":"NSDI"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/3132747.3132758"},{"key":"e_1_2_1_5_1","first-page":"1026","article-title":"Costbased query transformation in Oracle","volume":"6","author":"Ahmed Rafi","year":"2006","unstructured":"Rafi Ahmed, Allison Lee, Andrew Witkowski, Dinesh Das, Hong Su, Mohamed Zait, and Thierry Cruanes. 2006. Costbased query transformation in Oracle. In VLDB, Vol. 6. 1026--1036.","journal-title":"VLDB"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDE.1993.344026"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1109\/TKDE.2017.2773607"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/342009.335420"},{"key":"e_1_2_1_9_1","unstructured":"AWS. 2023. AWS Latency Monitoring. https:\/\/www.cloudping.co\/grid."},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/1066157.1066171"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDE.2015.7113294"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.14778\/3415478.3415548"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.14778\/3352063.3352126"},{"volume-title":"Apache flink: Stream and batch processing in a single engine. The Bulletin of the Technical Committee on Data Engineering 38, 4","year":"2015","author":"Carbone Paris","key":"e_1_2_1_14_1","unstructured":"Paris Carbone, Asterios Katsifodimos, Stephan Ewen, Volker Markl, Seif Haridi, and Kostas Tzoumas. 2015. Apache flink: Stream and batch processing in a single engine. The Bulletin of the Technical Committee on Data Engineering 38, 4 (2015)."},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/1559845.1559955"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.14778\/3554821.3554832"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/3447786.3456238"},{"key":"e_1_2_1_18_1","unstructured":"Inc. ClickHouse. 2023. ClickHouse - open source distributed column-oriented DBMS. https:\/\/github.com\/ClickHouse\/ ClickHouse\/tree\/22.6."},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/1988842.1988850"},{"key":"e_1_2_1_20_1","unstructured":"Brian Cooper. 2010. Yahoo! Cloud Serving Benchmark. https:\/\/github.com\/brianfrankcooper\/YCSB."},{"key":"e_1_2_1_21_1","unstructured":"The Transaction Processing Council. 1992. TPC-H. http:\/\/www.tpc.org\/tpch\/."},{"key":"e_1_2_1_22_1","unstructured":"The Transaction Processing Council. 2014. TPC-C. http:\/\/www.tpc.org\/tpcc\/."},{"key":"e_1_2_1_23_1","unstructured":"The Transaction Processing Council. 2015. TPC-DS. http:\/\/www.tpc.org\/tpcds\/."},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDEW.2014.6818330"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.14778\/3231751.3231761"},{"key":"e_1_2_1_26_1","unstructured":"Science Direct. 2004. Real-Time Pricing. https:\/\/www.sciencedirect.com\/topics\/engineering\/real-time-pricing."},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/2588555.2588566"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/3183713.3190660"},{"volume-title":"From Star Schemas to Big Data: 20 Years of Data Warehouse Research. A comprehensive guide through the Italian database research over the last 25 years","year":"2017","author":"Golfarelli Matteo","key":"e_1_2_1_30_1","unstructured":"Matteo Golfarelli and Stefano Rizzi. 2017. From Star Schemas to Big Data: 20 Years of Data Warehouse Research. A comprehensive guide through the Italian database research over the last 25 years (2017), 93--107."},{"key":"e_1_2_1_31_1","unstructured":"Google. 2022. AlloyDB for PostgreSQL under the hood: Columnar engine. https:\/\/cloud.google.com\/blog\/products\/ databases\/alloydb-for-postgresql-columnar-engine."},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/191839.191879"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.14778\/3415478.3415535"},{"key":"e_1_2_1_34_1","unstructured":"Bert Hubert. 2023. tc(8) Linux manual page. https:\/\/man7.org\/linux\/man-pages\/man8\/tc.8.html."},{"volume-title":"Unistore: A modern approach to working with transactional and analytical data together in a single platform. https:\/\/www.snowflake.com\/workloads\/unistore\/.","year":"2023","author":"SnowFlake Inc.","key":"e_1_2_1_35_1","unstructured":"SnowFlake Inc. 2023. Unistore: A modern approach to working with transactional and analytical data together in a single platform. https:\/\/www.snowflake.com\/workloads\/unistore\/."},{"volume-title":"Query and Resource Optimizations: A Case for Breaking the Wall in Big Data Systems. arXiv preprint arXiv:1906.06590","year":"2019","author":"Jindal Alekh","key":"e_1_2_1_36_1","unstructured":"Alekh Jindal, Lalitha Viswanathan, and Konstantinos Karanasos. 2019. Query and Resource Optimizations: A Case for Breaking the Wall in Big Data Systems. arXiv preprint arXiv:1906.06590 (2019)."},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/276304.276315"},{"key":"e_1_2_1_38_1","unstructured":"The kernel development community. 2023. Control Groups. https:\/\/docs.kernel.org\/admin-guide\/cgroup-v1\/cgroups.html."},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/3035918.3064049"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/3035918.3064049"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDE.2015.7113373"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.14778\/2824032.2824071"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.14778\/3137765.3137767"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0306-4379(02)00021-2"},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1145\/3514221.3522565"},{"volume-title":"2007 IEEE 23rd International Conference on Data Engineering. IEEE, 26--35","year":"2006","author":"Li Quanzhong","key":"e_1_2_1_46_1","unstructured":"Quanzhong Li, Minglong Shao, Volker Markl, Kevin Beyer, Latha Colby, and Guy Lohman. 2006. Adaptively reordering joins during query execution. In 2007 IEEE 23rd International Conference on Data Engineering. IEEE, 26--35."},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDE53745.2022.00071"},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.14778\/3372716.3372719"},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1145\/3448016.3457562"},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1145\/3514221.3526148"},{"key":"e_1_2_1_51_1","unstructured":"MySQL. 2022. MySQL Heatwave. https:\/\/dev.mysql.com\/doc\/heatwave\/en\/heatwave-introduction.html."},{"volume-title":"Speculative execution in a distributed file system. ACM SIGOPS operating systems review 39, 5","year":"2005","author":"Nightingale Edmund B","key":"e_1_2_1_52_1","unstructured":"Edmund B Nightingale, Peter M Chen, and Jason Flinn. 2005. Speculative execution in a distributed file system. ACM SIGOPS operating systems review 39, 5 (2005), 191--205."},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1145\/3035918.3054784"},{"key":"e_1_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-10424-4_17"},{"key":"e_1_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.14778\/2536222.2536233"},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.1145\/3318464.3389783"},{"volume-title":"L-store: A realtime OLTP and OLAP system. arXiv preprint arXiv:1601.04084","year":"2016","author":"Sadoghi Mohammad","key":"e_1_2_1_57_1","unstructured":"Mohammad Sadoghi, Souvik Bhattacherjee, Bishwaranjan Bhattacharjee, and Mustafa Canim. 2016. L-store: A realtime OLTP and OLAP system. arXiv preprint arXiv:1601.04084 (2016)."},{"key":"e_1_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.1145\/3318464.3389757"},{"volume-title":"Real-time LSM-trees for HTAP workloads. arXiv preprint arXiv:2101.06801","year":"2021","author":"Saxena Hemant","key":"e_1_2_1_59_1","unstructured":"Hemant Saxena, Lukasz Golab, Stratos Idreos, and Ihab F Ilyas. 2021. Real-time LSM-trees for HTAP workloads. arXiv preprint arXiv:2101.06801 (2021)."},{"key":"e_1_2_1_60_1","doi-asserted-by":"publisher","DOI":"10.1145\/582095.582099"},{"volume-title":"15th {USENIX} Symposium on Operating Systems Design and Implementation ({OSDI} 21). 219--238.","author":"Shen Sijie","key":"e_1_2_1_61_1","unstructured":"Sijie Shen, Rong Chen, Haibo Chen, and Binyu Zang. 2021. Retrofitting High Availability Mechanism to Tame Hybrid Transaction\/Analytical Processing. In 15th {USENIX} Symposium on Operating Systems Design and Implementation ({OSDI} 21). 219--238."},{"key":"e_1_2_1_62_1","unstructured":"Inc. SingleStore. 2023. SingleStore: Real-Time Distributed SQL. https:\/\/www.singlestore.com\/."},{"key":"e_1_2_1_63_1","first-page":"62","article-title":"From Star to Snowflake to ERD: Comparing Data Warehouse Design Approaches","volume":"14","author":"Spenser T","year":"1999","unstructured":"T Spenser and T Loukas. 1999. From Star to Snowflake to ERD: Comparing Data Warehouse Design Approaches. Enterprise Systems Journal 14 (1999), 62--69.","journal-title":"Enterprise Systems Journal"},{"key":"e_1_2_1_64_1","first-page":"9478","article-title":"Apache kafka: Next generation distributed messaging system","volume":"3","author":"Me Thein Khin Me","year":"2014","unstructured":"Khin Me Me Thein. 2014. Apache kafka: Next generation distributed messaging system. International Journal of Scientific Engineering and Technology Research 3, 47 (2014), 9478--9483.","journal-title":"International Journal of Scientific Engineering and Technology Research"},{"key":"e_1_2_1_65_1","doi-asserted-by":"publisher","DOI":"10.4018\/jdwm.2009070101"},{"key":"e_1_2_1_66_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDE.2018.00156"},{"key":"e_1_2_1_67_1","doi-asserted-by":"publisher","DOI":"10.1145\/3589785"},{"key":"e_1_2_1_68_1","doi-asserted-by":"publisher","DOI":"10.14778\/3415478.3415553"},{"volume-title":"Proceedings of the 2020 USENIX Conference on Usenix Annual Technical Conference. 17--31","year":"2020","author":"Yao Ting","key":"e_1_2_1_69_1","unstructured":"Ting Yao, Yiwen Zhang, JiguangWan, Qiu Cui, gLiu Tang, Hong Jiang, Changsheng Xie, and Xubin He. 2020. MatrixKV: reducing write stalls and write amplification in LSM-tree based KV stores with a matrix container in NVM. In Proceedings of the 2020 USENIX Conference on Usenix Annual Technical Conference. 17--31."},{"key":"e_1_2_1_70_1","doi-asserted-by":"publisher","DOI":"10.1145\/2854006.2854012"}],"container-title":["Proceedings of the ACM on Management of Data"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3626750","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,8,7]],"date-time":"2024-08-07T18:27:42Z","timestamp":1723055262000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3626750"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,12,8]]},"references-count":69,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2023,12,8]]}},"alternative-id":["10.1145\/3626750"],"URL":"https:\/\/doi.org\/10.1145\/3626750","relation":{},"ISSN":["2836-6573"],"issn-type":[{"type":"electronic","value":"2836-6573"}],"subject":[],"published":{"date-parts":[[2023,12,8]]}}}
