Abstract
Ascertaining the feasibility of independent falsification or repetition of published results is vital to the scientific process, and replication or reproduction experiments are routinely performed in many disciplines. Unfortunately, such studies are only scarcely available in database research, with few papers dedicated to re-evaluating published results. In this paper, we conduct a case study on replicating and reproducing a study on schema evolution in embedded databases. We can exactly repeat the outcome for one out of four database applications studied, and come close in two further cases. By reporting results, efforts, and obstacles encountered, we hope to increase appreciation for the substantial efforts required to ensure reproducibility. By discussing minutiae details required to ascertain reproducible work, we argue that such important, but often ignored aspects of scientific work should receive more credit in the evaluation of future research.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
Such as in the VLDB (“pVLDB Reproducibility”) and SIGMOD communities (“ACM SIGMOD 2019 Reproducibility”, clickable links available in PDF).
- 2.
The SQL dialects reference at https://en.wikibooks.org/wiki/SQL_Dialects_Reference illustrates the richness of proprietary language constructs.
- 3.
Revision 16 only changes the MySQL schema declaration, and the original study reports a schema change in this revision. A peak in schema changes is reported for revision 35 (see Table 2), as switching from MySQL to SQLite schema declarations causes mysqldiff to recognize type changes. Since revision 35 only adds support for SQLite, with no schema changes for MySQL or PostgreSQL, we conclude that starting with revision 35, the authors analyzed the SQLite schema.
References
Abadi, D., Ailamaki, A., Andersen, D., Bailis, P., et al.: The seattle report on database research. SIGMOD Rec. 48(4), 44–53 (2020)
Abramatic, J.F., Di Cosmo, R., Zacchiroli, S.: Building the universal archive of source code. Commun. ACM 61(10), 29–31 (2018)
Bellahsene, Z., Bonifati, A., Rahm, E.: Schema Matching and Mapping, 1st edn. Springer, Cham (2011)
Bird, C., Menzies, T., Zimmermann, T.: The Art and Science of Analyzing Software Data, 1st edn. Morgan Kaufmann Publishers Inc., San Francisco (2015)
Collberg, C., Proebsting, T.A.: Repeatability in computer systems research. Commun. ACM 59(3), 62–69 (2016)
Curino, C.A., Moon, H.J., Zaniolo, C.: Graceful database schema evolution: the prism workbench. VLDB Endow. 1, 761–772 (2008)
Curino, C.A., Tanca, L., Moon, H.J., Zaniolo, C.: Schema evolution in Wikipedia: toward a web information system benchmark. In: Proceedings of ICEIS 2008 (2008)
Lin, D.Y., Neamtiu, I.: Collateral evolution of applications and databases. In: Proceedings of IWPSE-Evol 2009 (2009)
Manolescu, I., Afanasiev, L., Arion, A., Dittrich, J., et al.: The repeatability experiment of SIGMOD 2008. SIGMOD Rec. 37(1), 39–45 (2008)
Moon, H.J., Curino, C.A., Deutsch, A., Hou, C.Y., Zaniolo, C.: Managing and querying transaction-time databases under schema evolution. VLDB Endow. 1, 882–895 (2008)
Neamtiu, I., Lin, D.Y., Uddin, R.: Safe on-the-fly relational schema evolution. Technical report (2009)
Pawlik, M., Hütter, T., Kocher, D., Mann, W., Augsten, N.: A link is not enough - reproducibility of data. Datenbank-Spektrum 19(2), 107–115 (2019)
Qiu, D., Li, B., Su, Z.: An empirical analysis of the co-evolution of schema and code in database applications. In: Proceedings of ESEC/FSE 2013 (2013)
Sjøberg, D.: Quantifying schema evolution. Inf. Softw. Technol. 35(1), 35–44 (1993)
Vassiliadis, P., Zarras, A.V., Skoulis, I.: How is life for a table in an evolving relational schema? Birth, death and everything in between. In: Johannesson, P., Lee, M.L., Liddle, S.W., Opdahl, A.L., López, Ó.P. (eds.) ER 2015. LNCS, vol. 9381, pp. 453–466. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-25264-3_34
Wu, S., Neamtiu, I.: Schema evolution analysis for embedded databases. In: Proceedings of ICDE Workshops 2011 (2011)
Acknowledgements
We thank the authors of [16] for sharing parts of their analysis code, and their feedback on an earlier version of this report. Stefanie Scherzinger’s contribution, within the scope of project “NoSQL Schema Evolution und Big Data Migration at Scale”, is funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation)—grant number 385808805.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Braininger, D., Mauerer, W., Scherzinger, S. (2020). Replicability and Reproducibility of a Schema Evolution Study in Embedded Databases. In: Grossmann, G., Ram, S. (eds) Advances in Conceptual Modeling. ER 2020. Lecture Notes in Computer Science(), vol 12584. Springer, Cham. https://doi.org/10.1007/978-3-030-65847-2_19
Download citation
DOI: https://doi.org/10.1007/978-3-030-65847-2_19
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-65846-5
Online ISBN: 978-3-030-65847-2
eBook Packages: Computer ScienceComputer Science (R0)