iBet uBet web content aggregator. Adding the entire web to your favor.
iBet uBet web content aggregator. Adding the entire web to your favor.



Link to original content: https://unpaywall.org/10.1007/S10664-016-9486-9
Eye movements in software traceability link recovery | Empirical Software Engineering Skip to main content
Springer Nature Link
Log in

Eye movements in software traceability link recovery

  • Published:
Empirical Software Engineering Aims and scope Submit manuscript

Abstract

Information Retrieval (IR) approaches, such as Latent Semantic Indexing (LSI) and Vector Space Model (VSM), are commonly applied to recover software traceability links. Recently, an approach based on developers’ eye gazes was proposed to retrieve traceability links. This paper presents a comparative study on IR and eye-gaze based approaches. In addition, it reports on the possibility of using eye gaze links as an alternative benchmark in comparison to commits. The study conducted asked developers to perform bug-localization tasks on the open source subject system JabRef. The iTrace environment, which is an eye tracking enabled Eclipse plugin, was used to collect eye gaze data. During the data collection phase, an eye tracker was used to gather the source code entities (SCE’s), developers looked at while solving these tasks. We present an algorithm that uses the collected gaze dataset to produce candidate traceability links related to the tasks. In the evaluation phase, we compared the results of our algorithm with the results of an IR technique, in two different contexts. In the first context, precision and recall metric values are reported for both IR and eye gaze approaches based on commits. In the second context, another set of developers were asked to rate the candidate links from each of the two techniques in terms of how useful they were in fixing the bugs. The eye gaze approach outperforms standard LSI and VSM approaches and reports a 55 % precision and 67 % recall on average for all tasks when compared to how the developers actually fixed the bug. In the second context, the usefulness results show that links generated by our algorithm were considered to be significantly more useful (to fix the bug) than those of the IR technique in a majority of tasks. We discuss the implications of this radically different method of deriving traceability links. Techniques for feature location/bug localization are commonly evaluated on benchmarks formed from commits as is done in the evaluation phase of this study. Although, commits are a reasonable source, they only capture entities that were eventually changed to fix a bug or resolve a feature. We investigate another type of benchmark based on eye tracking data, namely links generated from the bug-localization tasks given to the developers in the data collection phase. The source code entities relevant to subjected bugs recommended from IR methods are evaluated on both commits and links generated from eye gaze. The results of the benchmarking phase show that the use of eye tracking could form an effective (complementary) benchmark and add another interesting perspective in the evaluation of bug-localization techniques.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Notes

  1. https://github.com/seresl/itrace-gazelink-algorithm

  2. http://www.csis.ysu.edu/~bsharif/itrace-IRvsGaze

  3. 3 https://code.google.com/p/semanticvectors/

References

  • Alhindawi N, Dragan N, Collard M, Maletic J (2013) Improving feature location by enhancing source code with stereotypes. In: 29Th IEEE international conference on software maintenance, ICSM 2013

  • Ali N, Sharafi Z, Guéhéneuc Y. G., Antoniol G (2012) An empirical study on requirements traceability using eye-tracking. In: ICSM, pp 191 – 200

  • Antoniol G, Canfora G, Casazza G, De Lucia A, Merlo E (2002) Recovering traceability links between code and documentation. IEEE Trans Softw Eng 28(10):970–983

    Article  Google Scholar 

  • Asuncion H, Asuncion A, Taylor RN (2010) Software traceability with topic modeling. In: 32Nd ACM/IEEE international conference on software engineering, vol. 1, pp. 95–104. ACM

  • Asuncion H, Francois F, Taylor RN (2007) An end-to-end industrial software traceability tool. In: 6Th ESEC/FSE, pp 115–124

  • Bavota G, Lucia AD, Oliveto R, Panichella A, Ricci F, Tortora G (2013) The role of artefact corpus in lsi-based traceability recovery. 2013 7th International Workshop on Traceability in Emerging Forms of Software Engineering (TEFSE) 0, 83–89. doi:10.1109/TEFSE.2013.6620160

  • Binkley D, Lawrie D (2011) Information retrieval applications in software maintenance and evolution. In: Encyclopedia of software engineering

  • Capobianco G, De Lucia A, Oliveto R, Panichella A, Panichella S (2009) On the role of nouns in ir-based traceability recovery. In: 17Th IEEE international conference on program comprehension, pp 148–157

  • Cleland-Huang J, Czauderna A, Dekhtyar A, Gotel O, Hayes JH, Keenan E, Leach G, Maletic JI, Poshyvanyk D, Shin Y, Zisman A, Antoniol G, Berenbach B, Egyed A, Maeder P (2011) Grand challenges, benchmarks, and tracelab: developing infrastructure for the software traceability community. In: 6th TEFSE. Panel, pp 17–23

  • Cleland-Huang J, Gotel O, Zisman A (2012) Software and systems traceability Spring-Verlag

  • Coest.org (2007) Center of excellence for software traceability. http://www.coest.org/index.php/grand-challenges9

  • Davis APH (1993) Software requirements: Objects, Functions and States. Prentice Hall

  • De Lucia A, Fasano F, Oliveto R, Tortora G (2006) Can information retrieval techniques effectively support traceability link recovery?. In: 14Th IEEE international conference on program comprehension (ICPC’06), pp 307–316

  • De Lucia A, Fasano F, Oliveto R, Tortora G (2007) Recovering traceability links in software artefact management systems using information retrieval methods. ACM (TOSEM) 16(4):13

    Article  Google Scholar 

  • De Lucia A, Oliveto R, Tortora G (2009) Assessing ir-based traceability recovery tools through controlled experiments. Empirical Softw Engg 14(1):57–92. doi:10.1007/s10664-008-9090-8

    Article  Google Scholar 

  • Deerwester S, Dumais ST, Furnas GW, Landauer TK, Harshman R (1990) Indexing by latent semantic analysis. J Am Soc Inf Sci 41(6):391–407

    Article  Google Scholar 

  • Dit B, Holtzhauer A, Poshyvanyk D, Kagdi H (2013) A dataset from change history to support evaluation of software maintenance tasks. In: Proceedings of the 10th Working Conference on Mining Software Repositories, MSR ’13, pp. 131–134. IEEE Press, Piscataway, NJ, USA. http://dl.acm.org/citation.cfm?id=2487085.2487114

  • Dit B, Moritz E, Poshyvanyk D (2012) A tracelab-based solution for creating, conducting, and sharing feature location experiments. In: Program Comprehension (ICPC), 2012 IEEE 20th International Conference on. doi:10.1109/ICPC.2012.6240489, pp 203–208

  • Dit B., Revelle M., Gethers M., Poshyvanyk D. (2013) Feature location in source code: a taxonomy and survey. J Softw: Evol Process 25(1):53–95. doi:10.1002/smr.567

    Google Scholar 

  • Dit B, Revelle M, Gethers M, Poshyvanyk D (2013) Feature location in source code: a taxonomy and survey. J Soft Maint Evo.: Res Pract 25(1):53–95

    Article  Google Scholar 

  • Duchowski AT (2003) Eye tracking methodology: Theory and practice. Springer-Verlag, London

    Book  MATH  Google Scholar 

  • Fritz T, Begel A, Müller S. C., Yigit-Elliott S, Züger M. (2014) Using psycho-physiological measures to assess task difficulty in software development. In: Proceedings of the 36th international conference on software engineering, ICSE 2014. ACM, New York, pp 402–413

  • Gethers M, Oliveto R, Poshyvanyk D, De Lucia A (2011) On integrating orthogonal information retrieval methods to improve traceability link recovery. In: 27Th IEEE international conference on software maintenance (ICSM’11), pp 133–142

  • Gotel O, Cleland-Huang J, Hayes JH, Zisman A, Egyed A, Grünbacher P., Antoniol G (2012) The quest for ubiquity: A roadmap for software and systems traceability research. In: 2012 20th IEEE International Requirements Engineering Conference (RE). doi:10.1109/RE.2012.6345841, pp 71–80

  • Hill E, Bacchelli A, Binkley D, Dit B, Lawrie D, Oliveto R (2013) Which feature location technique is better?. In: Proceedings of the 2013 IEEE international conference on software maintenance, ICSM’13. IEEE Computer Society, Washington, pp 408–411

  • Jinshui Wang Xin Peng ZX, Zhao W (2011) An exploratory study of feature location process: Distinct phases, recurring patterns, and elementary actions. In: Proceedings of the 27th IEEE international conference on software maintenance, ICSM’11, pp 213–222

  • Keenan E, Czauderna A, Leach G, Cleland-Huang J, Shin Y (2012) Tracelab: An experimental workbench for equipping researchers to innovate, synthesize, and comparatively evaluate traceability solutions. In: ICSE, p 4

  • Kersten M, Murphy G (2005) Mylar: a degree-of-interest model for ides. In: 4th international conference on aspect-oriented software development, pp 159–168

  • Kevic K, Walters B, Shaffer T, Sharif B, Fritz T, Shepherd DC (2015) Tracing software developers eyes and interactions for change tasks Proceedings of the 10th Joint Meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering

  • Lin J, Lin CC, Cleland-Huang J, Settimi R, Amaya J, Bedford G, Berenbach B, Khadra OB, Duan C, Zou X (2006) Poirot: A distributed tool supporting enterprise-wide automated traceability. 2014 IEEE 22nd International Requirements Engineering Conference (RE) 0, 363–364. doi:10.1109/RE.2006.48

  • Lucia AD, Oliveto R, Sgueglia P (2006) Incremental approach and user feedbacks: a silver bullet for traceability recovery. In: 2006 22nd IEEE International Conference on Software Maintenance. doi:10.1109/ICSM.2006.32, pp 299–309

  • Lukins SK, Kraft NA, Etzkorn LH (2010) Bug localization using latent dirichlet allocation. Inf Softw Technol 52(9):972–990

    Article  Google Scholar 

  • Mader P, Gotel O, Philippow I (2009) Motivation matters in the traceability trenches. In: RE 2009, pp 143–148

  • Manning C, Raghavan P, Schütze H. (2008) Introduction to information retrieval cambridge university press

  • Marcus A, Maletic JI (2003) Recovering documentation-to-source-code traceability links using latent semantic indexing. In: 25Th IEEE/ACM international conference on software engineering (ICSE’03), pp 125–137

  • Oliveto R, Gethers M, Poshyvanyk D, De Lucia A (2010) On the equivalence of information retrieval methods for automated traceability link recovery. In: 18Th IEEE international conference on program comprehension (ICPC), pp 68–71

  • Qusef A, Oliveto R, De Lucia A (2010) Recovering traceability links between unit tests and classes under test: an improved method. In: IEEE International conference on software maintenance, pp 1–10

  • Ramesh B, Jarke M (2001) Towards reference models for requirements traceability. IEEE TSE 27(1):58–93

    Google Scholar 

  • Rao S, Kak A (2011) Retrieval from software libraries for bug localization: a comparative study of generic and composite text models. In: Proceedings of the 8th working conference on mining software repositories, MSR ’11. ACM, New York, pp 43–52

  • Rodeghero P, McMillan C, McBurney PW, Bosch N, D’Mello S (2014) Improving automated source code summarization via an eye-tracking study of programmers. In: ICSE 2014. ACM, New York, pp 390–401

  • Scanniello G, Marcus A (2011) Clustering support for static concept location in source code. In: Program Comprehension (ICPC), 2011 IEEE 19th International Conference on. doi:10.1109/ICPC.2011.13, pp 1–10

  • Scanniello G, Marcus A, Pascale D (2015) Link analysis algorithms for static concept location: An empirical assessment. Empir Softw Eng 20(6):1666–1720. doi:10.1007/s10664-014-9327-7

    Article  Google Scholar 

  • Shaffer T, Wise JL, Walters B, Müller S.C., Falcone M, Sharif B (2015) itrace: Enabling eye tracking on software artifacts within the ide to support software engineering tasks. In: Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering, ESEC/FSE 2015, pp. 954–957. ACM, New York, NY, USA. doi:10.1145/2786805.2803188

  • Sharafi Z, Soh Z, Guéhéneuc Y. G. (2015) A systematic literature review on the usage of eye-tracking in software engineering Elsevier Journal of Information and Software Technology (IST)

  • Sharif B, Kagdi H (2011) On the use of eye tracking in software traceability. In: 6Th TEFSE, pp 67–70

  • Walters B, Falcone M, Shibble A, Sharif B (2013) Towards an eye-tracking enabled ide for software traceability tasks. In: 7Th TEFSE, pp 51–54

  • Walters B, Shaffer T, Sharif B, Kagdi H (2014) Capturing software traceability links from developers’ eye gazes. In: Proceedings of the 22Nd International Conference on Program Comprehension, ICPC 2014, pp. 201–204. ACM, New York, NY, USA. doi:10.1145/2597008.2597795

  • Wang J, Peng X, Xing Z, Zhao W (2013) Improving feature location practice with multi-faceted interactive exploration. In: Proceedings of the 2013 International Conference on Software Engineering, ICSE ’13, pp. 762–771. IEEE Press, Piscataway, NJ, USA. http://dl.acm.org/citation.cfm?id=2486788.2486888

  • Xing Z, Xue Y, Jarzabek S (2013) A large scale linux-kernel based benchmark for feature location research. In: Proceedings of the 2013 International Conference on Software Engineering, ICSE ’13, pp. 1311–1314. IEEE Press, Piscataway, NJ, USA. http://dl.acm.org/citation.cfm?id=2486788.2486992

  • Zhou J, Zhang H, Lo D (2012) Where should the bugs be fixed? - more accurate information retrieval-based bug localization based on bug reports. In: Proceedings of the 34th international conference on software engineering, ICSE ’12. IEEE Press, Piscataway, pp 14–24

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science and Information Systems, Youngstown State University, Youngstown, OH, 44555, USA

    Bonita Sharif, John Meinken & Timothy Shaffer

  2. Department of Electrical Engineering and Computer Science, Wichita State University, Wichita, KS, 67260, USA

    Huzefa Kagdi

Authors
  1. Bonita Sharif
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. John Meinken
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Timothy Shaffer
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Huzefa Kagdi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Bonita Sharif.

Additional information

Communicated by: Patrick Mäder, Rocco Oliveto and Andrian Marcus

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Sharif, B., Meinken, J., Shaffer, T. et al. Eye movements in software traceability link recovery. Empir Software Eng 22, 1063–1102 (2017). https://doi.org/10.1007/s10664-016-9486-9

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10664-016-9486-9

Keywords

Search

Navigation