iBet uBet web content aggregator. Adding the entire web to your favor.
iBet uBet web content aggregator. Adding the entire web to your favor.



Link to original content: https://unpaywall.org/10.1007/978-3-030-32489-6_14
Benefitting from the Grey Literature in Software Engineering Research | SpringerLink
Skip to main content
Springer Nature Link
Log in

Benefitting from the Grey Literature in Software Engineering Research

  • Chapter
  • First Online:
Contemporary Empirical Methods in Software Engineering

Abstract

Researchers generally place the most trust in peer-reviewed, published information, such as journals and conference papers. By contrast, software engineering (SE) practitioners typically do not have the time, access, or expertise to review and benefit from such publications. As a result, practitioners are more likely to turn to other sources of information that they trust, e.g., trade magazines, online blog posts, survey results, or technical reports, collectively referred to as grey literature (GL). Furthermore, practitioners also share their ideas and experiences as GL, which can serve as a valuable data source for research. While GL itself is not a new topic in SE, using, benefitting, and synthesizing knowledge from the GL in SE is a contemporary topic in empirical SE research and we are seeing that researchers are increasingly benefitting from the knowledge available within GL. The goal of this chapter is to provide an overview of GL in SE, together with insights on how SE researchers can effectively use and benefit from the knowledge and evidence available in the vast amount of GL.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 129.00
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 169.00
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 169.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

Similar content being viewed by others

References

  • Aberkane A (2018) Exploring ethics in requirements engineering. Master thesis, Utrecht University

    Google Scholar 

  • Adams RJ, Smart P, Huff AS (2016) Shades of grey: guidelines for working with the grey literature in systematic reviews for management and organizational studies. Int J Manage Rev 19:432–454

    Article  Google Scholar 

  • Augusto L, Bakker MR, Morel C et al (2010) Is ‘grey literature’ a reliable source of data to characterize soils at the scale of a region? A case study in a maritime pine forest in southwestern France. Eur J Soil Sci 61(6):807–822

    Article  Google Scholar 

  • Bajwa SS, Wang X, Nguyen Duc A et al (2017) “Failures” to be celebrated: an analysis of major pivots of software startups. Empir Softw Eng 22(5):2373–2408

    Article  Google Scholar 

  • Banks MA (2006) Towards a continuum of scholarship: the eventual collapse of the distinction between grey and non-grey literature. Publ Res Q 22(1):4–11

    Article  Google Scholar 

  • Bansal N, Chiang F, Koudas N et al (2007) Seeking stable clusters in the blogosphere. In: Proceedings of VLDB endowment, pp 806–817

    Google Scholar 

  • Bonato S (2018) Searching the grey literature: a handbook for searching reports, working papers, and other unpublished research. RL & Associates LLC

    Google Scholar 

  • Briand L (2012) Embracing the engineering side of software engineering. IEEE Softw 29(4):96–96

    Article  Google Scholar 

  • Brito G, Terra R, Valente MT (2018) Monorepos: a multivocal literature review. arXiv preprint arXiv:1810.09477

    Google Scholar 

  • Brooks F (1995) The mythical man-month: essays on software engineering. Pearson Education, London

    Google Scholar 

  • Calderón A, Ruiz M, O’Connor RV (2018) A multivocal literature review on serious games for software process standards education. Comput Stand Interfaces 57:36–48

    Article  Google Scholar 

  • Calefato F, Lanubile F, Novielli N (2018) How to ask for technical help? Evidence-based guidelines for writing questions on Stack Overflow. Inf Softw Technol 94:186–207

    Article  Google Scholar 

  • Chavez TA, Perrault AH, Reehling P et al (2007) The impact of grey literature in advancing global karst research: an information needs assessment for a globally distributed interdisciplinary community. Publ Res Q 23:3

    Article  Google Scholar 

  • Corporation ED (2018) Global developer population and demographic study 2018. https://evansdata.com/reports/viewRelease.php?reportID=9

  • Cruzes DS, Dybå T (2011) Recommended steps for thematic synthesis in software engineering. In: International symposium on empirical software engineering and measurement. IEEE, pp 275–284

    Google Scholar 

  • de Franca BBN, Helvio Jeronimo J, Travassos GH (2016) Characterizing DevOps by hearing multiple voices. In: Proceedings of the Brazilian symposium on software engineering, pp 53–62

    Google Scholar 

  • DeMarco T, Lister T (2013) Peopleware: productive projects and teams. Addison-Wesley, Boston

    Google Scholar 

  • Devanbu P, Zimmermann T, Bird C (2016) Belief & evidence in empirical software engineering. In: IEEE/ACM international conference on software engineering. IEEE, pp 108–119

    Google Scholar 

  • Efstathiou V, Chatzilenas C, Spinellis D (2018) Word embeddings for the software engineering domain. In: Proceedings of the international conference on mining software repositories. ACM, pp 38–41

    Google Scholar 

  • Elliott C (2019) Jinfo Blog: Garner additional research sources with grey literature. https://web.jinfo.com/go/blog/70203

  • Facca FM, Lanzi PL (2005) Mining interesting knowledge from weblogs: a survey. Data Knowl Eng 53(3):225–241

    Article  Google Scholar 

  • Farace DJ (1997) Rise of the phoenix: a review of new forms and exploitations of grey literature. Publ Res Q 13(2):69–76

    Article  Google Scholar 

  • Garousi V, Küçük B (2018) Smells in software test code: a survey of knowledge in industry and academia. J Syst Softw 138:52–81

    Article  Google Scholar 

  • Garousi V, Mäntylä MV (2016) When and what to automate in software testing? A multivocal literature review. Inf Softw Technol 76:92–117

    Article  Google Scholar 

  • Garousi V, Felderer M, Mäntylä MV (2016a) The need for multivocal literature reviews in software engineering: complementing systematic literature reviews with grey literature. In: International conference on evaluation and assessment in software engineering. Limmerick, pp 171–176

    Google Scholar 

  • Garousi V, Petersen K, Özkan B (2016b) Challenges and best practices in industry-academia collaborations in software engineering: a systematic literature review. Inf Softw Technol 79:106–127

    Article  Google Scholar 

  • Garousi V, Felderer M, Fernandes JM et al (2017a) Industry-academia collaborations in software engineering: an empirical analysis of challenges, patterns and anti-patterns in research projects. In: Proceedings of international conference on evaluation and assessment in software engineering, Karlskrona, pp 224–229

    Google Scholar 

  • Garousi V, Felderer M, Hacaloğlu T (2017b) Software test maturity assessment and test process improvement: a multivocal literature review. Inf Softw Technol 85:16–42

    Article  Google Scholar 

  • Garousi V, Borg M, Oivo M (2018) Cut to the chase: revisiting the relevance of software engineering research. arXiv preprint arXiv:1812.01395

    Google Scholar 

  • Garousi V, Felderer M, Mäntylä MV (2019) Guidelines for including grey literature and conducting multivocal literature reviews in software engineering. Inf Softw Technol 106:101–121

    Article  Google Scholar 

  • Giustini D, Thompson D (2010) Finding the hard to finds: searching for grey (gray) literature. https://blogs.ubc.ca/dean/2010/02/finding-the-hard-to-finds-searching-for-grey-gray-literature-2010/

  • Gruetze T, Krestel R, Naumann F (2016) Topic shifts in stackoverflow: ask it like Socrates. In: International conference on applications of natural language to information systems. Springer, pp 213–221

    Google Scholar 

  • Institute for Work & Health (2019) What researchers mean by... Grey literature. https://www.iwh.on.ca/what-researchers-mean-by/grey-literature

  • Inui K, Abe S, Hara K et al (2008) Experience mining: building a large-scale database of personal experiences and opinions from web documents. In: IEEE Computer Society, pp 314–321

    Google Scholar 

  • Kitchenham B, Charters S (2007) Guidelines for performing systematic literature reviews in software engineering. EBSE technical report. EBSE-2007-01

    Google Scholar 

  • Kitchenham BA, Dybå T, Jorgensen M (2004) Evidence-based software engineering. In: Proceedings of international conference on software engineering, pp 273–281

    Google Scholar 

  • Kitchenham B, Budgen D, Brereton P (2015) Evidence-based software engineering and systematic reviews. CRC, Boca Raton

    Book  Google Scholar 

  • Kulesovs I (2015) iOS applications testing. In: Proceedings of the international scientific and practical conference, pp 138–150

    Google Scholar 

  • Kurashima T, Tezuka T, Tanaka K (2006) Mining and visualizing local experiences from blog entries. In: International conference on database and expert systems applications. Springer, pp 213–222

    Google Scholar 

  • Kurashima T, Fujimura K, Okuda H (2009) Discovering association rules on experiences from large-scale blog entries. In: European conference on information retrieval, pp 546–553

    Google Scholar 

  • Lakshmanan G, Oberhofer M (2010) Knowledge discovery in the blogosphere: approaches and challenges. IEEE Internet Comput 14(2):24–32

    Article  Google Scholar 

  • Lawrence S, Giles CL (1999) Accessibility of information on the web. Nature 400:107

    Article  Google Scholar 

  • Lefebvre C, Manheimer E, Glanville J (2008) Searching for studies. In: Higgins JPT, Green S (eds) Cochrane handbook for systematic reviews of interventions. Wiley-Blackwell, Chichester

    Google Scholar 

  • Leitner P, Wittern E, Spillner J et al (2018) A mixed-method empirical study of function-as-a-service software development in industrial practice. J Syst Softw 149:340–359

    Google Scholar 

  • Luzi D (2000) Trends and evolution in the development of grey literature: a review. Int J Grey Lit 1(3):106–117

    Article  Google Scholar 

  • Lwakatare LE, Kuvaja P, Oivo M (2016) Relationship of DevOps to agile, lean and continuous deployment: a multivocal literature review study. In: Proceedings of international conference on product-focused software process improvement, pp 399–415

    Google Scholar 

  • MacDonald BH, Cordes RE, Wells PG (2007) Assessing the diffusion and impact of grey literature published by international intergovernmental scientific groups: the case of the Gulf of Maine council on the marine environment. In: Proceedings of the international conference on grey literature, pp 84–94

    Google Scholar 

  • MacLeod L, Storey M-A, Bergen A (2015) Code, camera, action: how software developers document and share program knowledge using YouTube. In: Proceedings of the IEEE international conference on program comprehension, pp 104–114

    Google Scholar 

  • Mäntylä MV, Smolander K (2016) Gamification of software testing – an MLR. In: International conference on product-focused software process improvement, pp 611–614

    Google Scholar 

  • Marsolek W, Cooper K, Farrell S et al (2018) The types, frequencies, and findability of disciplinary grey literature within prominent subject databases and academic institutional repositories. J Librariansh Sch Commun 6(1)

    Google Scholar 

  • McKimmie T, Szurmak J (2002) Beyond grey literature: how grey questions can drive research. J Agric Food Inf 4(2):71–79

    Article  Google Scholar 

  • Molléri JS, Petersen K, Mendes E (2016) Survey guidelines in software engineering: an annotated review. In: Proceedings of the ACM/IEEE international symposium on empirical software engineering and measurement, 58, pp 51–56

    Google Scholar 

  • Myrbakken H, Colomo-Palacios R (2017) DevSecOps: a multivocal literature review. In: Conference on software process improvement and capability determination, pp 17–29

    Google Scholar 

  • Pappas C, Williams I (2011) Grey literature: its emerging importance. J Hosp Librariansh 11:228

    Article  Google Scholar 

  • Park KC, Jeong Y, Myaeng SH (2010) Detecting experiences from weblogs. In: Proceedings of the annual meeting of the association for computational linguistics, pp 1464–1472

    Google Scholar 

  • Parnin C, Treude C (2011) Measuring API documentation on the web. ACM, New York, pp 25–30

    Google Scholar 

  • Parnin C, Treude C, Storey M-A (2013) Blogging developer knowledge: motivations, challenges, and future directions. In: IEEE international conference on program comprehension, pp 211–214

    Google Scholar 

  • Passos C, Braun AP, Cruzes DS et al (2011) Analyzing the impact of beliefs in software project practices. In: International symposium on empirical software engineering and measurement. IEEE, pp 444–452

    Google Scholar 

  • Patton MQ (2002) Qualitative research and evaluation methods. Sage, Thousand Oaks

    Google Scholar 

  • Petersen K, Vakkalanka S, Kuzniarz L (2015) Guidelines for conducting systematic mapping studies in software engineering: an update. Inf Softw Technol 64:1–18

    Article  Google Scholar 

  • Planning S (2002) The economic impacts of inadequate infrastructure for software testing. National Institute of Standards and Technology

    Google Scholar 

  • Putta A, Paasivaara M, Lassenius C (2018) Benefits and challenges of adopting the scaled agile framework (SAFe): preliminary results from a multivocal literature review. In: International conference on product-focused software process improvement. Springer, pp 334–351

    Google Scholar 

  • Rainer A (2017) Using argumentation theory to analyse software practitioners’ defeasible evidence, inference and belief. Inf Softw Technol 87:62–80

    Article  Google Scholar 

  • Rainer A, Williams A (2019) Using blog-like documents to investigate software practice: benefits, challenges, and research directions. J Softw Evol Process 31(11):e2197

    Google Scholar 

  • Rainer A, Hall T, Baddoo N (2003) Persuading developers to “buy into” software process improvement: a local opinion and empirical evidence. In: International symposium on empirical software engineering. IEEE, pp 326–335

    Google Scholar 

  • Raulamo P, Mäntylä MV, Garousi V (2017) Choosing the right test automation tool: a grey literature review. In: International conference on evaluation and assessment in software engineering. Karlskrona, pp 21–30

    Google Scholar 

  • Research Triangle Institute (2002) The economic impacts of inadequate infrastructure for software testing. American National Institute of Standards and Technology (NIST), Technical report 7007.011

    Google Scholar 

  • Rincon P (2010) Stricter checks for climate body. https://www.bbc.com/news/science-environment-11131897

  • Rosen C, Shihab E (2016) What are mobile developers asking about? A large scale study using stack overflow. Empir Softw Eng 21(3):1192–1223

    Article  Google Scholar 

  • Sadaqat M, Colomo-Palacios R, Knudsen LES (2018) Serverless computing: a multivocal literature review. In: Norwegian conference for organizations’ use of information technology (NOKOBIT)

    Google Scholar 

  • Sánchez-Gordón M, Colomo-Palacios R (2018) A multivocal literature review on the use of DevOps for e-learning systems. In: Proceedings of international conference on technological ecosystems for enhancing multiculturality, pp 883–888

    Google Scholar 

  • Sharma P, Savarimuthu BTR, Stanger N et al (2017) Investigating developers’ email discussions during decision-making in Python language evolution. In: Proceedings of international conference on evaluation and assessment in software engineering. Karlskrona, ACM, pp 286–291

    Google Scholar 

  • Sharp H, deSouza C, Dittrich Y (2010) Using ethnographic methods in software engineering research. In: ACM/IEEE international conference on software engineering, pp 491–492

    Google Scholar 

  • Soldani J, Tamburri DA, Van Den Heuvel W-J (2018) The pains and gains of microservices: a systematic grey literature review. J Syst Softw 146:215–232

    Article  Google Scholar 

  • Storey M-A, Singer L, Cleary B et al (2014) The (r) evolution of social media in software engineering. In: Proceedings of the on future of software engineering, ACM, pp 100–116

    Google Scholar 

  • Swanson R, Rahimtoroghi E, Corcoran T et al (2014) Identifying narrative clause types in personal stories. In: Proceedings of the annual meeting of the special interest group on discourse and dialogue, pp 171–180

    Google Scholar 

  • The Standish Group (2019) CHAOS report. https://www.standishgroup.com/outline

  • Thompson L (2001) Grey literature in engineering. Sci Technol Libr 19(3–4):57–73

    Article  Google Scholar 

  • Tom E, Aurum A, Vidgen R (2013) An exploration of technical debt. J Syst Softw 86(6):1498–1516

    Article  Google Scholar 

  • University of New England (2019) Grey literature. https://www.une.edu.au/library/support/eskills-plus/research-skills/grey-literature

  • Williams A, Rainer A (2017) Toward the use of blog articles as a source of evidence for software engineering research. In: Proceedings of the international conference on evaluation and assessment in software engineering, pp 280–285

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Queen’s University Belfast, Belfast, Northern Ireland, UK

    Vahid Garousi & Austen Rainer

  2. University of Innsbruck, Innsbruck, Austria

    Michael Felderer

  3. Blekinge Institute of Technology, Karlskrona, Sweden

    Michael Felderer

  4. University of Oulu, Oulu, Finland

    Mika V. Mäntylä

Authors
  1. Vahid Garousi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Michael Felderer
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mika V. Mäntylä
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Austen Rainer
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Vahid Garousi .

Editor information

Editors and Affiliations

  1. Institute of Computer Science, University of Innsbruck, Innsbruck, Austria

    Michael Felderer

  2. Systems Engineering and Computer Science, Federal University of Rio de Janeiro, Rio de Janeiro, Rio de Janeiro, Brazil

    Guilherme Horta Travassos

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Garousi, V., Felderer, M., Mäntylä, M.V., Rainer, A. (2020). Benefitting from the Grey Literature in Software Engineering Research. In: Felderer, M., Travassos, G. (eds) Contemporary Empirical Methods in Software Engineering. Springer, Cham. https://doi.org/10.1007/978-3-030-32489-6_14

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-32489-6_14

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-32488-9

  • Online ISBN: 978-3-030-32489-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation