{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2024,7,20]],"date-time":"2024-07-20T23:24:13Z","timestamp":1721517853486},"reference-count":93,"publisher":"MDPI AG","issue":"7","license":[{"start":{"date-parts":[[2021,3,27]],"date-time":"2021-03-27T00:00:00Z","timestamp":1616803200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"FCT - Funda\u00e7\u00e3o para a Ci\u00eancia e a Tecnologia","award":["POCI - 01-0145 - FEDER- 031581"]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"An emergent research area in software engineering and software reliability is the use of wearable biosensors to monitor the cognitive state of software developers during software development tasks. The goal is to gather physiologic manifestations that can be linked to error-prone scenarios related to programmers\u2019 cognitive states. In this paper we investigate whether electroencephalography (EEG) can be applied to accurately identify programmers\u2019 cognitive load associated with the comprehension of code with different complexity levels. Therefore, a controlled experiment involving 26 programmers was carried. We found that features related to Theta, Alpha, and Beta brain waves have the highest discriminative power, allowing the identification of code lines and demanding higher mental effort. The EEG results reveal evidence of mental effort saturation as code complexity increases. Conversely, the classic software complexity metrics do not accurately represent the mental effort involved in code comprehension. Finally, EEG is proposed as a reference, in particular, the combination of EEG with eye tracking information allows for an accurate identification of code lines that correspond to peaks of cognitive load, providing a reference to help in the future evaluation of the space and time accuracy of programmers\u2019 cognitive state monitored using wearable devices compatible with software development activities.<\/jats:p>","DOI":"10.3390\/s21072338","type":"journal-article","created":{"date-parts":[[2021,3,29]],"date-time":"2021-03-29T03:27:25Z","timestamp":1616988445000},"page":"2338","source":"Crossref","is-referenced-by-count":18,"title":["Can EEG Be Adopted as a Neuroscience Reference for Assessing Software Programmers\u2019 Cognitive Load?"],"prefix":"10.3390","volume":"21","author":[{"ORCID":"http:\/\/orcid.org\/0000-0003-2852-6285","authenticated-orcid":false,"given":"J\u00falio","family":"Medeiros","sequence":"first","affiliation":[{"name":"Department of Informatics Engineering, CISUC-Centre for Informatics and Systems of the University of Coimbra, University of Coimbra, P-3030-790 Coimbra, Portugal"}]},{"ORCID":"http:\/\/orcid.org\/0000-0003-1237-6964","authenticated-orcid":false,"given":"Ricardo","family":"Couceiro","sequence":"additional","affiliation":[{"name":"Department of Informatics Engineering, CISUC-Centre for Informatics and Systems of the University of Coimbra, University of Coimbra, P-3030-790 Coimbra, Portugal"}]},{"ORCID":"http:\/\/orcid.org\/0000-0002-5331-5829","authenticated-orcid":false,"given":"Gon\u00e7alo","family":"Duarte","sequence":"additional","affiliation":[{"name":"Department of Informatics Engineering, CISUC-Centre for Informatics and Systems of the University of Coimbra, University of Coimbra, P-3030-790 Coimbra, Portugal"}]},{"ORCID":"http:\/\/orcid.org\/0000-0002-9697-9991","authenticated-orcid":false,"given":"Jo\u00e3o","family":"Dur\u00e3es","sequence":"additional","affiliation":[{"name":"Department of Informatics Engineering, CISUC-Centre for Informatics and Systems of the University of Coimbra, University of Coimbra, P-3030-790 Coimbra, Portugal"},{"name":"Coimbra Polytechnic\u2014ISEC, R. Pedro Nunes, P-3030-199 Coimbra, Portugal"}]},{"ORCID":"http:\/\/orcid.org\/0000-0002-8996-1515","authenticated-orcid":false,"given":"Jo\u00e3o","family":"Castelhano","sequence":"additional","affiliation":[{"name":"ICNAS-Institute of Nuclear Sciences Applied to Health, University of Coimbra, P-3000-548 Coimbra, Portugal"},{"name":"CIBIT-Coimbra Institute for Biomedical Imaging and Translational Research, University of Coimbra, P-3000-548 Coimbra, Portugal"}]},{"ORCID":"http:\/\/orcid.org\/0000-0002-5620-2424","authenticated-orcid":false,"given":"Catarina","family":"Duarte","sequence":"additional","affiliation":[{"name":"ICNAS-Institute of Nuclear Sciences Applied to Health, University of Coimbra, P-3000-548 Coimbra, Portugal"},{"name":"CIBIT-Coimbra Institute for Biomedical Imaging and Translational Research, University of Coimbra, P-3000-548 Coimbra, Portugal"}]},{"ORCID":"http:\/\/orcid.org\/0000-0003-4364-6373","authenticated-orcid":false,"given":"Miguel","family":"Castelo-Branco","sequence":"additional","affiliation":[{"name":"ICNAS-Institute of Nuclear Sciences Applied to Health, University of Coimbra, P-3000-548 Coimbra, Portugal"},{"name":"CIBIT-Coimbra Institute for Biomedical Imaging and Translational Research, University of Coimbra, P-3000-548 Coimbra, Portugal"}]},{"ORCID":"http:\/\/orcid.org\/0000-0001-8146-4664","authenticated-orcid":false,"given":"Henrique","family":"Madeira","sequence":"additional","affiliation":[{"name":"Department of Informatics Engineering, CISUC-Centre for Informatics and Systems of the University of Coimbra, University of Coimbra, P-3030-790 Coimbra, Portugal"}]},{"ORCID":"http:\/\/orcid.org\/0000-0002-9847-0590","authenticated-orcid":false,"given":"Paulo","family":"de Carvalho","sequence":"additional","affiliation":[{"name":"Department of Informatics Engineering, CISUC-Centre for Informatics and Systems of the University of Coimbra, University of Coimbra, P-3030-790 Coimbra, Portugal"}]},{"ORCID":"http:\/\/orcid.org\/0000-0001-9396-1211","authenticated-orcid":false,"given":"C\u00e9sar","family":"Teixeira","sequence":"additional","affiliation":[{"name":"Department of Informatics Engineering, CISUC-Centre for Informatics and Systems of the University of Coimbra, University of Coimbra, P-3030-790 Coimbra, Portugal"}]}],"member":"1968","published-online":{"date-parts":[[2021,3,27]]},"reference":[{"key":"ref_1","unstructured":"McConnell, S.C. (2004). Code Complete: A Practical Handbook of Software Construction, Microsoft Press."},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Shah, S.M.A., Morisio, M., and Torchiano, M. (2012, January 19\u201320). The impact of process maturity on defect density. Proceedings of the 2012 ACM-IEEE International Symposium on Empirical Software Engineering and Measurement, Lund, Sweden.","DOI":"10.1145\/2372251.2372308"},{"key":"ref_3","first-page":"9","article-title":"Achieving CMMI Level 5 improvements with MBASE and the CeBASE method","volume":"15","author":"Boehm","year":"2002","journal-title":"CrossTalk"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Honda, N., and Yamada, S. (2012). Empirical Analysis for High Quality SW Development. Am. J. Oper. Res.","DOI":"10.4236\/ajor.2012.21004"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Zhang, H. (2009, January 20\u201326). An investigation of the relationships between lines of code and defects. Proceedings of the 2009 IEEE International Conference on Software Maintenance, Edmonton, AB, Canada.","DOI":"10.1109\/ICSM.2009.5306304"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Sandu, I., Salceanu, A., and Bejenaru, O. (2018, January 3\u20136). New approach of the Customer Defects per Lines of Code metric in Automotive SW Development applications. Proceedings of the XXII World Congress of the International Measurement Confederation (IMEKO 2018), Belfast, UK.","DOI":"10.1088\/1742-6596\/1065\/5\/052006"},{"key":"ref_7","unstructured":"Judge Business School, University of Cambridge (2020, April 05). Financial Content: Cambridge University Study states Software Bugs Cost Economy $312 Billion per Year. Available online: http:\/\/insight.jbs.cam.ac.uk\/2013\/financial-content-cambridge-university-study-states-software-bugs-cost-economy-312-billion-per-year\/."},{"key":"ref_8","unstructured":"McPeak, A. (2020, April 05). What\u2019s the True Cost of a Software Bug?. Available online: https:\/\/crossbrowsertesting.com\/blog\/development\/software-bug-cost\/."},{"key":"ref_9","unstructured":"Matteson, S. (2020, April 05). Report: Software Failure Caused $1.7 Trillion in Financial Losses in 2017. Available online: https:\/\/www.techrepublic.com\/article\/report-software-failure-caused-1-7-trillion-in-financial-losses-in-2017\/."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Reason, J. (1990). Human Error, Cambridge University Press.","DOI":"10.1017\/CBO9781139062367"},{"key":"ref_11","unstructured":"Huang, F., Liu, B., and Huang, B. (2012, January 26\u201328). A taxonomy system to identify human error causes for software defects. Proceedings of the 18th ISSAT International Conference on Reliability and Quality in Design, Boston, MA, USA."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Huang, F. (2017). Human Error Analysis in Software Engineering. Theory and Application on Cognitive Factors and Risk Management: New Trends and Procedures, IntechOpen.","DOI":"10.5772\/intechopen.68392"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"656","DOI":"10.1080\/001401398186829","article-title":"Physiological workload reactions to increasing levels of task difficulty","volume":"41","author":"Veltman","year":"1998","journal-title":"Ergonomics"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"563","DOI":"10.1111\/j.1469-8986.1976.tb00882.x","article-title":"Heart rate and respiratory responses as a function of task difficulty: The use of discriminant analysis in the selection of psychologically sensitive physiological responses","volume":"13","author":"Walter","year":"1976","journal-title":"Psychophysiology"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Pfleging, B., Fekety, D.K., Schmidt, A., and Kun, A.L. (2016, January 7\u201312). A model relating pupil diameter to mental workload and lighting conditions. Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, San Jose, CA, USA.","DOI":"10.1145\/2858036.2858117"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Parnin, C. (2011, January 22\u201324). Subvocalization-toward hearing the inner thoughts of developers. Proceedings of the 2011 IEEE 19th International Conference on Program Comprehension, Kingston, ON, Canada.","DOI":"10.1109\/ICPC.2011.49"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"410","DOI":"10.1109\/TITB.2009.2036164","article-title":"Discriminating stress from cognitive load using a wearable EDA device","volume":"14","author":"Setz","year":"2009","journal-title":"IEEE Trans. Inf. Technol. Biomed."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Kyriakou, K., Resch, B., Sagl, G., Petutschnig, A., Werner, C., Niederseer, D., Liedlgruber, M., Wilhelm, F.H., Osborne, T., and Pykett, J. (2019). Detecting moments of stress from measurements of wearable physiological sensors. Sensors, 19.","DOI":"10.3390\/s19173805"},{"key":"ref_19","unstructured":"Fritz, T., Begel, A., M\u00fcller, S.C., Yigit-Elliott, S., and Z\u00fcger, M. (June, January 31). Using psycho-physiological measures to assess task difficulty in software development. Proceedings of the 36th International Conference on Software Engineering, Hyderabad, India."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"M\u00fcller, S.C., and Fritz, T. (2016, January 14\u201322). Using (bio) metrics to predict code quality online. Proceedings of the 2016 IEEE\/ACM 38th International Conference on Software Engineering (ICSE), Austin, TX, USA.","DOI":"10.1145\/2884781.2884803"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Couceiro, R., Duarte, G., Dur\u00e3es, J., Castelhano, J., Duarte, C., Teixeira, C., Branco, M.C., Carvalho, P., and Madeira, H. (2019, January 25\u201331). Biofeedback augmented software engineering: Monitoring of programmers\u2019 mental effort. Proceedings of the 2019 IEEE\/ACM 41st International Conference on Software Engineering: New Ideas and Emerging Results (ICSE-NIER), Montreal, QC, Canada.","DOI":"10.1109\/ICSE-NIER.2019.00018"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Couceiro, R., Duarte, G., Dur\u00e3es, J., Castelhano, J., Duarte, C., Teixeira, C., Branco, M.C., Carvalho, P., and Madeira, H. (2019, January 24\u201327). Pupillography as indicator of programmers\u2019 mental effort and cognitive overload. Proceedings of the 2019 49th Annual IEEE\/IFIP International Conference on Dependable Systems and Networks (DSN), Portland, OR, USA.","DOI":"10.1109\/DSN.2019.00069"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Couceiro, R., Barbosa, R., Dur\u00e3es, J., Duarte, G., Castelhano, J., Duarte, C., Teixeira, C., Laranjeiro, N., Medeiros, J., and Carvalho, P. (2019, January 28\u201331). Spotting Problematic Code Lines using Nonintrusive Programmers\u2019 Biofeedback. Proceedings of the 2019 IEEE 30th International Symposium on Software Reliability Engineering (ISSRE), Berlin, Germany.","DOI":"10.1109\/ISSRE.2019.00019"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"256","DOI":"10.1093\/iwc\/iwt032","article-title":"Automatic cognitive load detection from face, physiology, task performance and fusion during affective interference","volume":"26","author":"Hussain","year":"2014","journal-title":"Interact. Comput."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Sharafi, Z., Shaffer, T., Sharif, B., and Gu\u00e9h\u00e9neuc, Y.G. (2015, January 1\u20134). Eye-tracking metrics in software engineering. Proceedings of the 2015 Asia-Pacific Software Engineering Conference (APSEC), New Delhi, India.","DOI":"10.1109\/APSEC.2015.53"},{"key":"ref_26","first-page":"34","article-title":"EEG based cognitive workload assessment for maximum efficiency","volume":"7","author":"Shriram","year":"2013","journal-title":"Int. Organ. Sci. Res. IOSR"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Lee, J.C., and Tan, D.S. (2006, January 15\u201318). Using a low-cost electroencephalograph for task classification in HCI research. Proceedings of the 19th Annual ACM Symposium on User Interface Software and Technology, Montreux, Switzerland.","DOI":"10.1145\/1166253.1166268"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"165","DOI":"10.1109\/TBCAS.2014.2316224","article-title":"Wireless and wearable EEG system for evaluating driver vigilance","volume":"8","author":"Lin","year":"2014","journal-title":"IEEE Trans. Biomed. Circuits Syst."},{"key":"ref_29","unstructured":"(2020, April 22). NASA Task Load Index, Available online: https:\/\/humansystems.arc.nasa.gov\/groups\/TLX\/."},{"key":"ref_30","unstructured":"Scalabrino, S., Bavota, G., Vendome, C., Linares-V\u00e1squez, M., Poshyvanyk, D., and Oliveto, R. (November, January 30). Automatically assessing code understandability: How far are we?. Proceedings of the 32nd IEEE\/ACM International Conference on Automated Software Engineering, Urbana, IL, USA."},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"McCabe, T.J. (1976). A complexity measure. IEEE Trans. Softw. Eng., 308\u2013320.","DOI":"10.1109\/TSE.1976.233837"},{"key":"ref_32","unstructured":"Halstead, M.H. (1977). Elements of Software Science, Elsevier."},{"key":"ref_33","unstructured":"Kasto, N., and Whalley, J. (February, January 29). Measuring the difficulty of code comprehension tasks using software metrics. Proceedings of the Fifteenth Australasian Computing Education Conference, Adelaide, Australia."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"812","DOI":"10.1007\/s10664-011-9162-z","article-title":"Calculation and optimization of thresholds for sets of software metrics","volume":"16","author":"Herbold","year":"2011","journal-title":"Empir. Softw. Eng."},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Minelli, R., Mocci, A., and Lanza, M. (2015, January 18\u201319). I know what you did last summer: An investigation of how developers spend their time. Proceedings of the 2015 IEEE 23rd International Conference on Program Comprehension, Florence, Italy.","DOI":"10.1109\/ICPC.2015.12"},{"key":"ref_36","unstructured":"Siegmund, J., K\u00e4stner, C., Apel, S., Parnin, C., Bethmann, A., Leich, T., Saake, G., and Brechmann, A. (June, January 31). Understanding understanding source code with functional magnetic resonance imaging. Proceedings of the 36th International Conference on Software Engineering, Hyderabad, India."},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Floyd, B., Santander, T., and Weimer, W. (2017, January 20\u201328). Decoding the representation of code in the brain: An fMRI study of code review and expertise. Proceedings of the 2017 IEEE\/ACM 39th International Conference on Software Engineering (ICSE), Buenos Aires, Argentina.","DOI":"10.1109\/ICSE.2017.24"},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Siegmund, J., Peitek, N., Parnin, C., Apel, S., Hofmeister, J., K\u00e4stner, C., Begel, A., Bethmann, A., and Brechmann, A. (2017, January 4\u20138). Measuring neural efficiency of program comprehension. Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering, Paderborn, Germany.","DOI":"10.1145\/3106237.3106268"},{"key":"ref_39","unstructured":"Peitek, N., Siegmund, J., Apel, S., K\u00e4stner, C., Parnin, C., Bethmann, A., Leich, T., Saake, G., and Brechmann, A. (2018). A Look into Programmers\u2019 Heads. IEEE Trans. Softw. Eng."},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Peitek, N., Siegmund, J., Parnin, C., Apel, S., Hofmeister, J.C., and Brechmann, A. (2018, January 11\u201312). Simultaneous measurement of program comprehension with fMRI and eye tracking: A case study. Proceedings of the 12th ACM\/IEEE International Symposium on Empirical Software Engineering and Measurement, Oulu, Finland.","DOI":"10.1145\/3239235.3240495"},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Peitek, N., Siegmund, J., Parnin, C., Apel, S., and Brechmann, A. (2018, January 14\u201317). Toward conjoint analysis of simultaneous eye-tracking and fMRI data for program-comprehension studies. Proceedings of the Workshop on Eye Movements in Programming, Warsaw, Poland.","DOI":"10.1145\/3216723.3216725"},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"623","DOI":"10.1007\/s11682-018-9885-1","article-title":"The role of the insula in intuitive expert bug detection in computer code: An fMRI study","volume":"13","author":"Castelhano","year":"2019","journal-title":"Brain Imaging Behav."},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Krueger, R., Huang, Y., Liu, X., Santander, T., Weimer, W., and Leach, K. (2020, January 6\u201311). Neurological Divide: An fMRI Study of Prose and Code Writing. Proceedings of the 2020 IEEE\/ACM 42nd International Conference on Software Engineering (ICSE), Seoul, Korea.","DOI":"10.1145\/3377811.3380348"},{"key":"ref_44","unstructured":"Nakagawa, T., Kamei, Y., Uwano, H., Monden, A., Matsumoto, K., and German, D.M. (June, January 31). Quantifying programmers\u2019 mental workload during program comprehension based on cerebral blood flow measurement: A controlled experiment. Proceedings of the Companion Proceedings of the 36th International Conference on Software Engineering, Hyderabad, India."},{"key":"ref_45","unstructured":"Ikutani, Y., and Uwano, H. (July, January 30). Brain activity measurement during program comprehension with NIRS. Proceedings of the 15th IEEE\/ACIS International Conference on Software Engineering, Artificial Intelligence, Networking and Parallel\/Distributed Computing (SNPD), Las Vegas, NV, USA."},{"key":"ref_46","first-page":"15","article-title":"Studies of Emotion: A Theoretical and Empirical Review of Psychophysiological Studies of Emotion","volume":"1","author":"Niemic","year":"2002","journal-title":"JUR Rochester"},{"key":"ref_47","doi-asserted-by":"crossref","unstructured":"Malmivuo, P., Malmivuo, J., and Plonsey, R. (1995). Electroencephalography. Bioelectromagnetism: Principles and Applications of Bioelectric and Biomagnetic Fields, Oxford University Press. Chapter 13.","DOI":"10.1093\/acprof:oso\/9780195058239.001.0001"},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"38","DOI":"10.1111\/j.1528-1157.1999.tb00905.x","article-title":"Mental Effort-Related EEG Modulation During Video-Game Play: Comparison Between Juvenile Subjects with Epilepsy and Normal Control Subjects","volume":"40","author":"Pellouchoud","year":"1999","journal-title":"Epilepsia"},{"key":"ref_49","doi-asserted-by":"crossref","unstructured":"Smith, M.E., and Gevins, A. (2005). Neurophysiologic monitoring of mental workload and fatigue during operation of a flight simulator. Biomonitoring for Physiological and Cognitive Performance during Military Operations, International Society for Optics and Photonics.","DOI":"10.1117\/12.602181"},{"key":"ref_50","first-page":"B231","article-title":"EEG correlates of task engagement and mental workload in vigilance, learning, and memory tasks","volume":"78","author":"Berka","year":"2007","journal-title":"Aviat. Space, Environ. Med."},{"key":"ref_51","doi-asserted-by":"crossref","first-page":"79","DOI":"10.1518\/001872098779480578","article-title":"Monitoring working memory load during computer-based tasks with EEG pattern recognition methods","volume":"40","author":"Gevins","year":"1998","journal-title":"Hum. Factors"},{"key":"ref_52","doi-asserted-by":"crossref","first-page":"169","DOI":"10.1016\/S0165-0173(98)00056-3","article-title":"EEG alpha and theta oscillations reflect cognitive and memory performance: A review and analysis","volume":"29","author":"Klimesch","year":"1999","journal-title":"Brain Res. Rev."},{"key":"ref_53","doi-asserted-by":"crossref","unstructured":"Crk, I., and Kluthe, T. (2014, January 26\u201330). Toward using alpha and theta brain waves to quantify programmer expertise. Proceedings of the 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, Chicago, IL, USA.","DOI":"10.1109\/EMBC.2014.6944840"},{"key":"ref_54","doi-asserted-by":"crossref","unstructured":"Lee, S., Matteson, A., Hooshyar, D., Kim, S., Jung, J., Nam, G., and Lim, H. (November, January 31). Comparing programming language comprehension between novice and expert programmers using eeg analysis. Proceedings of the 2016 IEEE 16th International Conference on Bioinformatics and Bioengineering (BIBE), Taichung, Taiwan.","DOI":"10.1109\/BIBE.2016.30"},{"key":"ref_55","doi-asserted-by":"crossref","first-page":"1097","DOI":"10.1007\/s10586-017-0746-2","article-title":"Mining biometric data to predict programmer expertise and task difficulty","volume":"21","author":"Lee","year":"2018","journal-title":"Clust. Comput."},{"key":"ref_56","doi-asserted-by":"crossref","unstructured":"Yeh, M.K.C., Gopstein, D., Yan, Y., and Zhuang, Y. (2017, January 18\u201321). Detecting and comparing brain activity in short program comprehension using EEG. Proceedings of the 2017 IEEE Frontiers in Education Conference (FIE), Indianapolis, IN, USA.","DOI":"10.1109\/FIE.2017.8190486"},{"key":"ref_57","doi-asserted-by":"crossref","first-page":"52","DOI":"10.1016\/j.ijhcs.2018.03.002","article-title":"Towards an affordable brain computer interface for the assessment of programmers\u2019 mental workload","volume":"115","author":"Kosti","year":"2018","journal-title":"Int. J. Hum. Comput. Stud."},{"key":"ref_58","doi-asserted-by":"crossref","first-page":"98","DOI":"10.1109\/JBHI.2017.2688239","article-title":"DREAMER: A database for emotion recognition through EEG and ECG signals from wireless low-cost off-the-shelf devices","volume":"22","author":"Katsigiannis","year":"2017","journal-title":"IEEE J. Biomed. Health Inform."},{"key":"ref_59","unstructured":"Hart, S.G., and Staveland, L.E. Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. Advances in Psychology, Elsevier."},{"key":"ref_60","doi-asserted-by":"crossref","first-page":"797","DOI":"10.1109\/32.879815","article-title":"Quantitative analysis of faults and failures in a complex software system","volume":"26","author":"Fenton","year":"2000","journal-title":"IEEE Trans. Softw. Eng."},{"key":"ref_61","doi-asserted-by":"crossref","first-page":"9","DOI":"10.1016\/j.jneumeth.2003.10.009","article-title":"EEGLAB: An open source toolbox for analysis of single-trial EEG dynamics including independent component analysis","volume":"134","author":"Delorme","year":"2004","journal-title":"J. Neurosci. Methods"},{"key":"ref_62","doi-asserted-by":"crossref","first-page":"184","DOI":"10.1016\/0013-4694(89)90180-6","article-title":"Spherical splines for scalp potential and current density mapping","volume":"72","author":"Perrin","year":"1989","journal-title":"Electroencephalogr. Clin. Neurophysiol."},{"key":"ref_63","unstructured":"Cohen, M.X. (2014). Preprocessing Steps Necessary and Useful for Advanced Data Analysis. Analyzing Neural Time Series Data: Theory and Practice, MIT Press. Chapter 7."},{"key":"ref_64","unstructured":"Cohen, M.X. (2014). EEG Artifacts: Their Detection, Influence, and Removal. Analyzing Neural Time Series Data: Theory and Practice, MIT Press. Chapter 8."},{"key":"ref_65","doi-asserted-by":"crossref","first-page":"417","DOI":"10.1162\/089976699300016719","article-title":"Independent component analysis using an extended infomax algorithm for mixed subgaussian and supergaussian sources","volume":"11","author":"Lee","year":"1999","journal-title":"Neural Comput."},{"key":"ref_66","doi-asserted-by":"crossref","first-page":"031001","DOI":"10.1088\/1741-2560\/12\/3\/031001","article-title":"EEG artifact removal\u2014State-of-the-art and guidelines","volume":"12","year":"2015","journal-title":"J. Neural Eng."},{"key":"ref_67","unstructured":"Alarcao, S.M., and Fonseca, M.J. (2017). Emotions recognition using EEG signals: A survey. IEEE Trans. Affect. Comput."},{"key":"ref_68","doi-asserted-by":"crossref","first-page":"306","DOI":"10.1016\/0013-4694(70)90143-4","article-title":"EEG analysis based on time domain properties","volume":"29","author":"Hjorth","year":"1970","journal-title":"Electroencephalogr. Clin. Neurophysiol."},{"key":"ref_69","doi-asserted-by":"crossref","first-page":"327","DOI":"10.1109\/TAFFC.2014.2339834","article-title":"Feature extraction and selection for emotion recognition from EEG","volume":"5","author":"Jenke","year":"2014","journal-title":"IEEE Trans. Affect. Comput."},{"key":"ref_70","doi-asserted-by":"crossref","first-page":"187","DOI":"10.1016\/0301-0511(95)05116-3","article-title":"Biocybernetic system evaluates indices of operator engagement in automated task","volume":"40","author":"Pope","year":"1995","journal-title":"Biol. Psychol."},{"key":"ref_71","doi-asserted-by":"crossref","first-page":"283","DOI":"10.1016\/j.biopsycho.2004.01.002","article-title":"An evaluation of an adaptive automation system using a cognitive vigilance task","volume":"67","author":"Freeman","year":"2004","journal-title":"Biol. Psychol."},{"key":"ref_72","unstructured":"Lei, S. (2011). Driver Mental States Monitoring Based on Brain Signals. [Ph.D. Thesis, TU Berlin]."},{"key":"ref_73","doi-asserted-by":"crossref","first-page":"340","DOI":"10.1007\/BF02359037","article-title":"Frequency characteristics of EEG spectra in the emotions","volume":"26","author":"Kostyunina","year":"1996","journal-title":"Neurosci. Behav. Physiol."},{"key":"ref_74","doi-asserted-by":"crossref","first-page":"110","DOI":"10.1080\/13854040600744839","article-title":"EEG neurofeedback: A brief overview and an example of peak alpha frequency training for cognitive enhancement in the elderly","volume":"21","author":"Angelakis","year":"2007","journal-title":"Clin. Neuropsychol."},{"key":"ref_75","doi-asserted-by":"crossref","first-page":"639","DOI":"10.1100\/tsw.2009.83","article-title":"Estimating brain load from the EEG","volume":"9","author":"Holm","year":"2009","journal-title":"Sci. World J."},{"key":"ref_76","doi-asserted-by":"crossref","first-page":"154","DOI":"10.1016\/j.intcom.2012.04.003","article-title":"A survey of methods for data fusion and system adaptation using autonomic nervous system responses in physiological computing","volume":"24","author":"Novak","year":"2012","journal-title":"Interact. Comput."},{"key":"ref_77","doi-asserted-by":"crossref","first-page":"94","DOI":"10.1016\/j.neucom.2013.06.046","article-title":"Emotional state classification from EEG data using machine learning approach","volume":"129","author":"Wang","year":"2014","journal-title":"Neurocomputing"},{"key":"ref_78","doi-asserted-by":"crossref","first-page":"583","DOI":"10.1080\/01621459.1952.10483441","article-title":"Use of ranks in one-criterion variance analysis","volume":"47","author":"Kruskal","year":"1952","journal-title":"J. Am. Stat. Assoc."},{"key":"ref_79","doi-asserted-by":"crossref","unstructured":"Mann, H.B., and Whitney, D.R. (1947). On a test of whether one of two random variables is stochastically larger than the other. Ann. Math. Stat., 50\u201360.","DOI":"10.1214\/aoms\/1177730491"},{"key":"ref_80","doi-asserted-by":"crossref","first-page":"23","DOI":"10.1023\/A:1025667309714","article-title":"Theoretical and empirical analysis of ReliefF and RReliefF","volume":"53","author":"Kononenko","year":"2003","journal-title":"Mach. Learn."},{"key":"ref_81","unstructured":"Kononenko, I., Robnik-Sikonja, M., and Pompe, U. (1996). ReliefF for estimation and discretization of attributes in classification, regression, and ILP problems. Artificial Intelligence: Methodology, Systems, Applications, IOS Press."},{"key":"ref_82","first-page":"69","article-title":"A guide to appropriate use of correlation coefficient in medical research","volume":"24","author":"Mukaka","year":"2012","journal-title":"Malawi Med. J."},{"key":"ref_83","doi-asserted-by":"crossref","first-page":"189","DOI":"10.1109\/TNN.2008.2005601","article-title":"Normalized mutual information feature selection","volume":"20","author":"Tesmer","year":"2009","journal-title":"IEEE Trans. Neural Netw."},{"key":"ref_84","first-page":"20150202","article-title":"Principal component analysis: A review and recent developments","volume":"374","author":"Jolliffe","year":"2016","journal-title":"Philos. Trans. R. Soc. Math. Phys. Eng. Sci."},{"key":"ref_85","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.chemolab.2015.10.003","article-title":"A MATLAB toolbox for Principal Component Analysis and unsupervised exploration of data structure","volume":"149","author":"Ballabio","year":"2015","journal-title":"Chemom. Intell. Lab. Syst."},{"key":"ref_86","first-page":"1","article-title":"Linear discriminant analysis-a brief tutorial","volume":"18","author":"Balakrishnama","year":"1998","journal-title":"Inst. Signal Inf. Process."},{"key":"ref_87","doi-asserted-by":"crossref","first-page":"121","DOI":"10.1023\/A:1009715923555","article-title":"A tutorial on support vector machines for pattern recognition","volume":"2","author":"Burges","year":"1998","journal-title":"Data Min. Knowl. Discov."},{"key":"ref_88","doi-asserted-by":"crossref","first-page":"21","DOI":"10.1109\/TIT.1967.1053964","article-title":"Nearest neighbor pattern classification","volume":"13","author":"Cover","year":"1967","journal-title":"IEEE Trans. Inf. Theory"},{"key":"ref_89","unstructured":"Rish, I. (2001, January 4\u201310). An empirical study of the naive Bayes classifier. Proceedings of the IJCAI 2001 Workshop on Empirical Methods in Artificial Intelligence, Seattle, WA, USA."},{"key":"ref_90","doi-asserted-by":"crossref","first-page":"40","DOI":"10.1214\/09-SS054","article-title":"A survey of cross-validation procedures for model selection","volume":"4","author":"Arlot","year":"2010","journal-title":"Stat. Surv."},{"key":"ref_91","doi-asserted-by":"crossref","first-page":"695","DOI":"10.1080\/10447318.2018.1461763","article-title":"Evaluating photoplethysmogram as a real-time cognitive load assessment during game playing","volume":"34","author":"Zhang","year":"2018","journal-title":"Int. J. Hum. Comput. Interact."},{"key":"ref_92","unstructured":"Bianchi, A.M. (2019, January 26\u201328). Efficacy of Time-and Frequency-Domain Heart Rate Variability Features in Stress Detection and Their Relation with Coping Strategies. Proceedings of the XV Mediterranean Conference on Medical and Biological Engineering and Computing\u2013MEDICON 2019, Coimbra, Portugal."},{"key":"ref_93","doi-asserted-by":"crossref","first-page":"276","DOI":"10.1037\/0033-2909.91.2.276","article-title":"Task-evoked pupillary responses, processing load, and the structure of processing resources","volume":"91","author":"Beatty","year":"1982","journal-title":"Psychol. Bull."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/7\/2338\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,7,10]],"date-time":"2024-07-10T06:11:02Z","timestamp":1720591862000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/7\/2338"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,3,27]]},"references-count":93,"journal-issue":{"issue":"7","published-online":{"date-parts":[[2021,4]]}},"alternative-id":["s21072338"],"URL":"https:\/\/doi.org\/10.3390\/s21072338","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,3,27]]}}}
