{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2024,8,30]],"date-time":"2024-08-30T11:39:38Z","timestamp":1725017978176},"reference-count":31,"publisher":"Association for Computing Machinery (ACM)","issue":"4","license":[{"start":{"date-parts":[[2020,10,12]],"date-time":"2020-10-12T00:00:00Z","timestamp":1602460800000},"content-version":"vor","delay-in-days":366,"URL":"http:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/501100001823","name":"Czech Ministry of Education, Youth and Sports","doi-asserted-by":"crossref","award":["CZ.02.1.010.00.015_0030000421"],"id":[{"id":"10.13039\/501100001823","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/100000001","name":"NSF","doi-asserted-by":"publisher","award":["1518844, 1544542, and 1617892"],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]},{"name":"European Research Council under the European Union's Horizon 2020 research and innovation programme","award":["695412"]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Program. Lang. Syst."],"published-print":{"date-parts":[[2019,12,31]]},"abstract":"In a 2014 article, Ray, Posnett, Devanbu, and Filkov claimed to have uncovered a statistically significant association between 11 programming languages and software defects in 729 projects hosted on GitHub. Specifically, their work answered four research questions relating to software defects and programming languages. With data and code provided by the authors, the present article first attempts to conduct an experimental repetition of the original study. The repetition is only partially successful, due to missing code and issues with the classification of languages. The second part of this work focuses on their main claim, the association between bugs and languages, and performs a complete, independent reanalysis of the data and of the statistical modeling steps undertaken by Ray et al. in 2014. This reanalysis uncovers a number of serious flaws that reduce the number of languages with an association with defects down from 11 to only 4. Moreover, the practical effect size is exceedingly small. These results thus undermine the conclusions of the original study. Correcting the record is important, as many subsequent works have cited the 2014 article and have asserted, without evidence, a causal link between the choice of programming language for a given task and the number of software defects. Causation is not supported by the data at hand; and, in our opinion, even after fixing the methodological flaws we uncovered, too many unaccounted sources of bias remain to hope for a meaningful comparison of bug rates across languages.<\/jats:p>","DOI":"10.1145\/3340571","type":"journal-article","created":{"date-parts":[[2019,10,15]],"date-time":"2019-10-15T16:35:58Z","timestamp":1571157358000},"page":"1-24","update-policy":"http:\/\/dx.doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":42,"title":["On the Impact of Programming Languages on Code Quality"],"prefix":"10.1145","volume":"41","author":[{"given":"Emery D.","family":"Berger","sequence":"first","affiliation":[{"name":"University of Massachusetts Amherst and Microsoft Research"}]},{"given":"Celeste","family":"Hollenbeck","sequence":"additional","affiliation":[{"name":"Northeastern University"}]},{"given":"Petr","family":"Maj","sequence":"additional","affiliation":[{"name":"Czech Technical University in Prague"}]},{"given":"Olga","family":"Vitek","sequence":"additional","affiliation":[{"name":"Northeastern University"}]},{"ORCID":"http:\/\/orcid.org\/0000-0003-4052-3458","authenticated-orcid":false,"given":"Jan","family":"Vitek","sequence":"additional","affiliation":[{"name":"Northeastern University and Czech Technical University in Prague"}]}],"member":"320","published-online":{"date-parts":[[2019,10,12]]},"reference":[{"key":"e_1_2_2_1_1","doi-asserted-by":"publisher","DOI":"10.2307\/2346101"},{"key":"e_1_2_2_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/1595696.1595716"},{"key":"e_1_2_2_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/3092703.3098230"},{"key":"e_1_2_2_4_1","doi-asserted-by":"publisher","DOI":"10.1098\/rsos.171085"},{"key":"e_1_2_2_5_1","unstructured":"Premkumar T. Devanbu. 2018. Research Statement. Retrieved from www.cs.ucdavis.edu\/∼devanbu\/research.pdf."},{"key":"e_1_2_2_6_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2013.6606588"},{"key":"e_1_2_2_7_1","volume-title":"Extending the Linear Model with R: Generalized Linear, Mixed Effects and Nonparametric Regression Models","author":"Faraway J. J.","unstructured":"J. J. Faraway. 2016. Extending the Linear Model with R: Generalized Linear, Mixed Effects and Nonparametric Regression Models. CRC Press."},{"key":"e_1_2_2_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/2723872.2723875"},{"key":"e_1_2_2_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/1852786.1852790"},{"key":"e_1_2_2_10_1","unstructured":"Garrett Grolemund and Hadley Wickham. 2017. R for Data Science. O\u2019Reilly."},{"key":"e_1_2_2_11_1","doi-asserted-by":"publisher","DOI":"10.1038\/nmeth.3288"},{"key":"e_1_2_2_12_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2013.6606585"},{"key":"e_1_2_2_13_1","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pmed.0020124"},{"key":"e_1_2_2_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/3243734.3243804"},{"key":"e_1_2_2_15_1","volume-title":"Functional languages rack up best scores for software quality. InfoWorld (Nov","author":"Krill Paul","year":"2014","unstructured":"Paul Krill. 2014. Functional languages rack up best scores for software quality. InfoWorld (Nov. 2014). https:\/\/www.infoworld.com\/article\/2844268\/functional-languages-rack-up-best-scores-software-quality.html."},{"key":"e_1_2_2_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/2658987"},{"key":"e_1_2_2_17_1","volume-title":"Applied Linear Statistical Models. McGraw\u2013Hill Education","author":"Kutner Michael H.","unstructured":"Michael H. Kutner, John Neter, Christopher J. Nachtsheim, and William Li. 2004. Applied Linear Statistical Models. McGraw\u2013Hill Education, New York, NY. https:\/\/books.google.cz\/books?id=XAzYCwAAQBAJ"},{"key":"e_1_2_2_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/3133908"},{"key":"e_1_2_2_19_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSM.2000.883028"},{"key":"e_1_2_2_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/2568225.2568324"},{"key":"e_1_2_2_21_1","volume-title":"Proceedings of the International Conference on Software Engineering (ICSE\u201915)","author":"Nanz Sebastian","year":"1875","unstructured":"Sebastian Nanz and Carlo A. Furia. 2015. A comparative study of programming languages in rosetta code. In Proceedings of the International Conference on Software Engineering (ICSE\u201915). http:\/\/dl.acm.org\/citation.cfm?id=2818754.2818848."},{"key":"e_1_2_2_22_1","doi-asserted-by":"publisher","DOI":"10.1126\/science.1213847"},{"key":"e_1_2_2_23_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2016.10.017"},{"key":"e_1_2_2_24_1","doi-asserted-by":"publisher","unstructured":"B. Ray and D. Posnett. 2016. A large ecosystem study to understand the effect of programming languages on code quality. In Perspectives on Data Science for Software Engineering. Morgan Kaufmann. DOI:https:\/\/doi.org\/10.1016\/B978-0-12-804206-9.00023-4","DOI":"10.1016\/B978-0-12-804206-9.00023-4"},{"key":"e_1_2_2_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/3126905"},{"key":"e_1_2_2_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/2635868.2635922"},{"key":"e_1_2_2_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/3180155.3180161"},{"key":"e_1_2_2_28_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2012.6227176"},{"key":"e_1_2_2_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/2038642.2038650"},{"key":"e_1_2_2_30_1","doi-asserted-by":"publisher","DOI":"10.1080\/00031305.2016.1154108"},{"key":"e_1_2_2_31_1","volume-title":"How does bug-handling effort differ among different programming languages? CoRR abs\/1801.01025","author":"Zhang Jie","year":"2018","unstructured":"Jie Zhang, Feng Li, Dan Hao, Meng Wang, and Lu Zhang. 2018. How does bug-handling effort differ among different programming languages? CoRR abs\/1801.01025 (2018). http:\/\/arxiv.org\/abs\/1801.01025."}],"container-title":["ACM Transactions on Programming Languages and Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3340571","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3340571","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,2,9]],"date-time":"2024-02-09T23:16:22Z","timestamp":1707520582000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3340571"}},"subtitle":["A Reproduction Study"],"short-title":[],"issued":{"date-parts":[[2019,10,12]]},"references-count":31,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2019,12,31]]}},"alternative-id":["10.1145\/3340571"],"URL":"https:\/\/doi.org\/10.1145\/3340571","relation":{},"ISSN":["0164-0925","1558-4593"],"issn-type":[{"value":"0164-0925","type":"print"},{"value":"1558-4593","type":"electronic"}],"subject":[],"published":{"date-parts":[[2019,10,12]]},"assertion":[{"value":"2018-12-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2019-06-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2019-10-12","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}
