iBet uBet web content aggregator. Adding the entire web to your favor.

Link to original content: https://doi.org/10.1145/3510003.3510169

FADATest | Proceedings of the 44th International Conference on Software Engineering

research-article

FADATest: fast and adaptive performance regression testing of dynamic binary translation systems

Authors:

Decheng ZuoAuthors Info & Claims

ICSE '22: Proceedings of the 44th International Conference on Software Engineering

Pages 896 - 908

https://doi.org/10.1145/3510003.3510169

Published: 05 July 2022 Publication History

Abstract

Dynamic binary translation (DBT) is the cornerstone of many important applications. In practice, however, it is quite difficult to maintain the performance efficiency of a DBT system due to its inherent complexity. Although performance regression testing is an effective approach to detect potential performance regression issues, it is not easy to apply performance regression testing to DBT systems, because of the natural differences between DBT systems and common software systems and the limited availability of effective test programs. In this paper, we present FADATest, which devises several novel techniques to address these challenges. Specifically, FADATest automatically generates adaptable test programs from existing real benchmark programs of DBT systems according to the runtime characteristics of the benchmarks. The test programs can then be used to achieve highly efficient and adaptive performance regression testing of DBT systems. We have implemented a prototype of FADATest. Experimental results show that FADATest can successfully uncover the same performance regression issues across the evaluated versions of two popular DBT systems, QEMU and Valgrind, as the original benchmark programs. Moreover, the testing efficiency is improved significantly on two different hardware platforms powered by x86-64 and AArch64, respectively.

References

[1]

Adam Henson. 2019. Automatic Website Performance Regression Testing. https://www.freecodecamp.org/news/automatic-website-performance-regression-testing-4e30e6bf5cd.

[2]

Mojtaba Bagherzadeh, Nafiseh Kahani, Cor-Paul Bezemer, Ahmed E Hassan, Juergen Dingel, and James R Cordy. 2018. Analyzing a decade of Linux system calls. Empirical Software Engineering 23, 3 (2018), 1519--1551.

Digital Library

[3]

Fabrice Bellard. 2005. QEMU, a Fast and Portable Dynamic Translator. In Proceedings of the Annual Conference on USENIX Annual Technical Conference (Anaheim, CA) (ATC '05). USENIX Association, USA, 41.

Digital Library

[4]

Abderrahmane Benbachir, Isnaldo Francisco De Melo, Michel Dagenais, and Bram Adams. 2017. Automated Performance Deviation Detection across Software Versions Releases. In 2017 IEEE International Conference on Software Quality, Reliability and Security (QRS). IEEE, 450--457.

[5]

Christian Bienia, Sanjeev Kumar, Jaswinder Pal Singh, and Kai Li. 2008. The PARSEC Benchmark Suite: Characterization and Architectural Implications. In Proceedings of the 17th International Conference on Parallel Architectures and Compilation Techniques (Toronto, Ontario, Canada) (PACT '08). Association for Computing Machinery, New York, NY, USA, 72--81.

Digital Library

[6]

David Brumley, Ivan Jager, Thanassis Avgerinos, and Edward J. Schwartz. 2011. BAP: A Binary Analysis Platform. In Proceedings of the 23rd International Conference on Computer Aided Verification (Snowbird, UT) (CAV'11). Springer-Verlag, Berlin, Heidelberg, 463--469.

[7]

Jacob Burnim, Sudeep Juvekar, and Koushik Sen. 2009. WISE: Automated test generation for worst-case complexity. In 2009 IEEE 31st International Conference on Software Engineering. IEEE, 463--473.

Digital Library

[8]

Jie Chen, Dongjin Yu, Haiyang Hu, Zhongjin Li, and Hua Hu. 2019. Analyzing Performance-Aware Code Changes in Software Development Process. In Proceedings of the 27th International Conference on Program Comprehension (Montreal, Quebec, Canada) (ICPC '19). IEEE Press, 300--310.

Digital Library

[9]

Andrea Fioraldi, Daniele Cono D'Elia, and Emilio Coppa. 2020. WEIZZ: Automatic Grey-Box Fuzzing for Structured Binary Formats. In Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis (Virtual Event, USA) (ISSTA 2020). Association for Computing Machinery, New York, NY, USA, 1--13.

Digital Library

[10]

King Chun Foo, Zhen Ming Jiang, Bram Adams, Ahmed E Hassan, Ying Zou, and Parminder Flora. 2015. An industrial case study on the automated detection of performance regressions in heterogeneous environments. In 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering, Vol. 2. IEEE, 159--168.

[11]

Gunnar Morling. 2020. Towards Continuous Performance Regression Testing. https://www.morling.dev/blog/towards-continuous-performance-regression-testing.

[12]

Greg Hamerly, Erez Perelman, and Brad Calder. 2004. How to use simpoint to pick simulation points. ACM SIGMETRICS Performance Evaluation Review 31, 4 (2004), 25--30.

Digital Library

[13]

Christoph Heger, Jens Happe, and Roozbeh Farahbod. 2013. Automated Root Cause Isolation of Performance Regressions during Software Development. In Proceedings of the 4th ACM/SPEC International Conference on Performance Engineering (Prague, Czech Republic) (ICPE '13). Association for Computing Machinery, New York, NY, USA, 27--38.

Digital Library

[14]

Peng Huang, Xiao Ma, Dongcai Shen, and Yuanyuan Zhou. 2014. Performance Regression Testing Target Prioritization via Performance Risk Analysis. In Proceedings of the 36th International Conference on Software Engineering (Hyderabad, India) (ICSE '14). Association for Computing Machinery, New York, NY, USA, 60--71.

Digital Library

[15]

Jinhu Jiang, Rongchao Dong, Zhongjun Zhou, Changheng Song, Wenwen Wang, Pen-Chung Yew, and Weihua Zhang. 2020. More with Less - Deriving More Translation Rules with Less Training Data for DBTs Using Parameterization. In 2020 53rd Annual IEEE/ACM International Symposium on Microarchitecture (MICRO). 415--426.

[16]

Ivo Jimenez, Noah Watkins, Michael Sevilla, Jay Lofstead, and Carlos Maltzahn. 2018. Quiho: Automated Performance Regression Testing Using Inferred Resource Utilization Profiles. In Proceedings of the 2018 ACM/SPEC International Conference on Performance Engineering (Berlin, Germany) (ICPE '18). Association for Computing Machinery, New York, NY, USA, 273--284.

Digital Library

[17]

Joel Beales and Jeffrey Dunn. 2018. MobileLab: Highly accurate testing to prevent mobile performance regressions. https://engineering.fb.com/2018/10/19/android/mobilelab.

[18]

Tomas Kalibera, Lubomir Bulej, and Petr Tuma. 2005. Automated detection of performance regressions: The Mono experience. In 13th IEEE International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems. IEEE, 183--190.

Digital Library

[19]

Rafaqut Kazmi, Dayang NA Jawawi, Radziah Mohamad, and Imran Ghani. 2017. Effective regression test case selection: A systematic literature review. ACM Computing Surveys (CSUR) 50, 2 (2017), 1--32.

Digital Library

[20]

Muhammad Khatibsyarbini, Mohd Adham Isa, Dayang NA Jawawi, and Rooster Tumeng. 2018. Test case prioritization approaches in regression testing: A systematic literature review. Information and Software Technology 93 (2018), 74--93.

Digital Library

[21]

Stephen Kyle, Igor Böhm, Björn Franke, Hugh Leather, and Nigel Topham. 2012. Efficiently Parallelizing Instruction Set Simulation of Embedded Multi-Core Processors Using Region-Based Just-in-Time Dynamic Binary Translation. In Proceedings of the 13th ACM SIGPLAN/SIGBED International Conference on Languages, Compilers, Tools and Theory for Embedded Systems (Beijing, China) (LCTES '12). Association for Computing Machinery, New York, NY, USA, 21--30.

Digital Library

[22]

Philipp Leitner and Cor-Paul Bezemer. 2017. An Exploratory Study of the State of Practice of Performance Testing in Java-Based Open Source Projects. In Proceedings of the 8th ACM/SPEC on International Conference on Performance Engineering (L'Aquila, Italy) (ICPE '17). Association for Computing Machinery, New York, NY, USA, 373--384.

Digital Library

[23]

Yuekang Li, Bihuan Chen, Mahinthan Chandramohan, Shang-Wei Lin, Yang Liu, and Alwen Tiu. 2017. Steelix: Program-State Based Binary Fuzzing. In Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering (Paderborn, Germany) (ESEC/FSE 2017). Association for Computing Machinery, New York, NY, USA, 627--637.

Digital Library

[24]

Qi Luo, Denys Poshyvanyk, and Mark Grechanik. 2016. Mining Performance Regression Inducing Code Changes in Evolving Software (MSR '16). Association for Computing Machinery, New York, NY, USA, 25--36.

Digital Library

[25]

Qi Luo, Denys Poshyvanyk, Aswathy Nair, and Mark Grechanik. 2016. FORE-POST: A Tool for Detecting Performance Problems with Feedback-Driven Learning Software Testing. In Proceedings of the 38th International Conference on Software Engineering Companion (Austin, Texas) (ICSE '16). Association for Computing Machinery, New York, NY, USA, 593--596.

Digital Library

[26]

Shaikh Mostafa, Xiaoyin Wang, and Tao Xie. 2017. PerfRanker: Prioritization of Performance Regression Tests for Collection-Intensive Software. In Proceedings of the 26th ACM SIGSOFT International Symposium on Software Testing and Analysis (Santa Barbara, CA, USA) (ISSTA 2017). Association for Computing Machinery, New York, NY, USA, 23--34.

Digital Library

[27]

Thanh H.D. Nguyen, Bram Adams, Zhen Ming Jiang, Ahmed E. Hassan, Mohamed Nasser, and Parminder Flora. 2012. Automated Detection of Performance Regressions Using Statistical Process Control Techniques. In Proceedings of the 3rd ACM/SPEC International Conference on Performance Engineering (Boston, Massachusetts, USA) (ICPE '12). Association for Computing Machinery, New York, NY, USA, 299--310.

Digital Library

[28]

Michael Pradel, Markus Huggler, and Thomas R Gross. 2014. Performance regression testing of concurrent classes. In Proceedings of the 2014 International Symposium on Software Testing and Analysis. 13--25.

Digital Library

[29]

QEMU wiki. 2021. Projects using the QEMU code. https://wiki.qemu.org/Links#Projects_using_the_QEMU_code.

[30]

David Georg Reichelt and Stefan Kühne. 2018. How to Detect Performance Changes in Software History: Performance Analysis of Software System Versions. In Companion of the 2018 ACM/SPEC International Conference on Performance Engineering (Berlin, Germany) (ICPE '18). Association for Computing Machinery, New York, NY, USA, 183--188.

Digital Library

[31]

Gregg Rothermel and Mary Jean Harrold. 1996. Analyzing regression test selection techniques. IEEE Transactions on software engineering 22, 8 (1996), 529--551.

Digital Library

[32]

Daniel Sanchez and Christos Kozyrakis. 2013. ZSim: Fast and Accurate Microarchitectural Simulation of Thousand-Core Systems. In Proceedings of the 40th Annual International Symposium on Computer Architecture (Tel-Aviv, Israel) (ISCA '13). Association for Computing Machinery, New York, NY, USA, 475--486.

Digital Library

[33]

Dawn Song, David Brumley, Heng Yin, Juan Caballero, Ivan Jager, Min Gyung Kang, Zhenkai Liang, James Newsome, Pongsin Poosankam, and Prateek Saxena. 2008. BitBlaze: A New Approach to Computer Security via Binary Analysis. In Proceedings of the 4th International Conference on Information Systems Security (Hyderabad, India) (ICISS '08). Springer-Verlag, Berlin, Heidelberg, 1--25.

Digital Library

[34]

Standard Performance Evaluation Corporation. 2020. SPEC CPU 2017. https://www.spec.org/cpu2017.

[35]

Amanda Swearngin, Myra B. Cohen, Bonnie E. John, and Rachel K. E. Bellamy. 2013. Human Performance Regression Testing. In Proceedings of the 2013 International Conference on Software Engineering (San Francisco, CA, USA) (ICSE '13). IEEE Press, 152--161.

[36]

Wenwen Wang. 2021. Helper Function Inlining in Dynamic Binary Translation. In Proceedings of the 30th ACM SIGPLAN International Conference on Compiler Construction. Association for Computing Machinery, New York, NY, USA, 107--118.

Digital Library

[37]

Wenwen Wang, Pen-Chung Yew, Antonia Zhai, and Stephen McCamant. 2020. Efficient and Scalable Cross-ISA Virtualization of Hardware Transactional Memory. In Proceedings of the 18th ACM/IEEE International Symposium on Code Generation and Optimization. Association for Computing Machinery, New York, NY, USA, 107--120.

Digital Library

[38]

Wenwen Wang, Pen-Chung Yew, Antonia Zhai, Stephen McCamant, Youfeng Wu, and Jayaram Bobba. 2017. Enabling Cross-ISA Offloading for COTS Binaries. In Proceedings of the 15th Annual International Conference on Mobile Systems, Applications, and Services (Niagara Falls, New York, USA) (MobiSys '17). Association for Computing Machinery, New York, NY, USA, 319--331.

Digital Library

[39]

Jin Wu, Jian Dong, Ruili Fang, Wenwen Wang, and Decheng Zuo. 2020. PerfDBT: Efficient Performance Regression Testing of Dynamic Binary Translation. In 2020 IEEE 38th International Conference on Computer Design (ICCD). 389--392.

[40]

Jin Wu, Jian Dong, Ruili Fang, Ziyi Zhao, Xiaoli Gong, Wenwen Wang, and Decheng Zuo. 2021. Effective Exploitation of SIMD Resources in Cross-ISA Virtualization. In Proceedings of the 17th ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments (Virtual, USA) (VEE 2021). Association for Computing Machinery, New York, NY, USA, 84--97.

Digital Library

[41]

Ziyi Zhao, Zhang Jiang, Ying Chen, Xiaoli Gong, Wenwen Wang, and Pen-Chung Yew. 2021. Enhancing Atomic Instruction Emulation for Cross-ISA Dynamic Binary Translation. In 2021 IEEE/ACM International Symposium on Code Generation and Optimization (CGO). 351--362.

Digital Library

[42]

Ziyi Zhao, Zhang Jiang, Ximing Liu, Xiaoli Gong, Wenwen Wang, and Pen-Chung Yew. 2020. DQEMU: A Scalable Emulator with Retargetable DBT on Distributed Platforms. In 49th International Conference on Parallel Processing - ICPP (Edmonton, AB, Canada) (ICPP '20). Association for Computing Machinery, New York, NY, USA, Article 7, 11 pages.

Digital Library

Cited By

Sun LWu YLi LZhang CTang J(2023)A Dynamic and Static Binary Translation Method Based on Branch PredictionElectronics10.3390/electronics1214302512:14(3025)Online publication date: 10-Jul-2023
https://doi.org/10.3390/electronics12143025
Laurent TArcaini PTrubiani CVentresque ARoychoudhury ACadar CKim M(2022)JSIMutate: understanding performance results through mutationsProceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3540250.3558930(1721-1725)Online publication date: 7-Nov-2022
https://dl.acm.org/doi/10.1145/3540250.3558930

Index Terms

FADATest: fast and adaptive performance regression testing of dynamic binary translation systems
1. Software and its engineering
  1. Software notations and tools
    1. Compilers
      1. Source code generation
  2. Software organization and properties
    1. Extra-functional properties
      1. Software performance
    2. Software system structures
      1. Software architectures
        Simulator / interpreter

Recommendations

Mining Performance Regression Testing Repositories for Automated Performance Analysis
QSIC '10: Proceedings of the 2010 10th International Conference on Quality Software

Performance regression testing detects performance regressions in a system under load. Such regressions refer to situations where software performance degrades compared to previous releases, although the new version behaves correctly. In current ...
Multiple-Core under Test Architecture for HOY Wireless Testing Platform
ATS '09: Proceedings of the 2009 Asian Test Symposium

Test integration for heterogeneous cores under test has been a challenging problem in a system-on-chip (SoC) design. To integrate heterogeneous cores under test, the test wrapper should be capable of dealing with multiple-clock domain problems, at-speed ...
Software-based self-test of processors under power constraints
DATE '06: Proceedings of the conference on Design, automation and test in Europe: Proceedings

Software-based self-test (SBST) of processors offers many benefits, such as dispense with expensive test equipments, test execution during maintenance and in the field or initialization tests for the whole system. In this paper, for the first time a ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ICSE '22: Proceedings of the 44th International Conference on Software Engineering

May 2022

2508 pages

ISBN:9781450392211

DOI:10.1145/3510003

General Chair:
Matthew B Dwyer
University of Virginia
,
Program Chairs:
Daniela Damian
University of Victoria, Canada
,
Andreas Zeller
CISPA, Germany

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering

In-Cooperation

IEEE CS

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 05 July 2022

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

University of Georgia

Conference

ICSE '22

Sponsor:

SIGSOFT

ICSE '22: 44th International Conference on Software Engineering

May 21 - 29, 2022

Pennsylvania, Pittsburgh

Acceptance Rates

Overall Acceptance Rate 276 of 1,856 submissions, 15%

Upcoming Conference

ICSE 2025

2025 IEEE/ACM 46th International Conference on Software Engineering

April 26 - May 3, 2025

Ottawa , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
151
Total Downloads

Downloads (Last 12 months)53
Downloads (Last 6 weeks)7

Reflects downloads up to 04 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Sun LWu YLi LZhang CTang J(2023)A Dynamic and Static Binary Translation Method Based on Branch PredictionElectronics10.3390/electronics1214302512:14(3025)Online publication date: 10-Jul-2023
https://doi.org/10.3390/electronics12143025
Laurent TArcaini PTrubiani CVentresque ARoychoudhury ACadar CKim M(2022)JSIMutate: understanding performance results through mutationsProceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3540250.3558930(1721-1725)Online publication date: 7-Nov-2022
https://dl.acm.org/doi/10.1145/3540250.3558930

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents