{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2024,5,31]],"date-time":"2024-05-31T00:28:48Z","timestamp":1717115328347},"reference-count":57,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2024,5,30]],"date-time":"2024-05-30T00:00:00Z","timestamp":1717027200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Robot. AI"],"abstract":"Introduction:<\/jats:bold> Flow state, the optimal experience resulting from the equilibrium between perceived challenge and skill level, has been extensively studied in various domains. However, its occurrence in industrial settings has remained relatively unexplored. Notably, the literature predominantly focuses on Flow within mentally demanding tasks, which differ significantly from industrial tasks. Consequently, our understanding of emotional and physiological responses to varying challenge levels, specifically in the context of industry-like tasks, remains limited.<\/jats:p>Methods:<\/jats:bold> To bridge this gap, we investigate how facial emotion estimation (valence, arousal) and Heart Rate Variability (HRV) features vary with the perceived challenge levels during industrial assembly tasks. Our study involves an assembly scenario that simulates an industrial human-robot collaboration task with three distinct challenge levels. As part of our study, we collected video, electrocardiogram (ECG), and NASA-TLX questionnaire data from 37 participants.<\/jats:p>Results:<\/jats:bold> Our results demonstrate a significant difference in mean arousal and heart rate between the low-challenge (Boredom) condition and the other conditions. We also found a noticeable trend-level difference in mean heart rate between the adaptive (Flow) and high-challenge (Anxiety) conditions. Similar differences were also observed in a few other temporal HRV features like Mean NN and Triangular index. Considering the characteristics of typical industrial assembly tasks, we aim to facilitate Flow by detecting and balancing the perceived challenge levels. Leveraging our analysis results, we developed an HRV-based machine learning model for discerning perceived challenge levels, distinguishing between low and higher-challenge conditions.<\/jats:p>Discussion:<\/jats:bold> This work deepens our understanding of emotional and physiological responses to perceived challenge levels in industrial contexts and provides valuable insights for the design of adaptive work environments.<\/jats:p>","DOI":"10.3389\/frobt.2024.1393795","type":"journal-article","created":{"date-parts":[[2024,5,30]],"date-time":"2024-05-30T08:40:30Z","timestamp":1717058430000},"update-policy":"http:\/\/dx.doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["Flow in human-robot collaboration\u2014multimodal analysis and perceived challenge detection in industrial scenarios"],"prefix":"10.3389","volume":"11","author":[{"given":"Pooja","family":"Prajod","sequence":"first","affiliation":[]},{"given":"Matteo","family":"Lavit Nicora","sequence":"additional","affiliation":[]},{"given":"Marta","family":"Mondellini","sequence":"additional","affiliation":[]},{"given":"Matteo Meregalli","family":"Falerni","sequence":"additional","affiliation":[]},{"given":"Rocco","family":"Vertechy","sequence":"additional","affiliation":[]},{"given":"Matteo","family":"Malosio","sequence":"additional","affiliation":[]},{"given":"Elisabeth","family":"Andr\u00e9","sequence":"additional","affiliation":[]}],"member":"1965","published-online":{"date-parts":[[2024,5,30]]},"reference":[{"key":"B1","doi-asserted-by":"publisher","first-page":"5","DOI":"10.1016\/j.cirp.2010.03.043","article-title":"Assessment of operator stress induced by robot collaboration in assembly","volume":"59","author":"Arai","year":"2010","journal-title":"CIRP Ann."},{"key":"B2","unstructured":"Blazeface: sub-millisecond neural face detection on mobile gpus\n BazarevskyV.\n KartynnikY.\n VakunovA.\n RaveendranK.\n GrundmannM.\n 2019"},{"key":"B3","first-page":"1","article-title":"Socially interactive agents as cobot avatars: developing a model to support flow experiences and weil-being in the workplace","author":"Beyrodt","year":"2023"},{"key":"B4","doi-asserted-by":"publisher","first-page":"821","DOI":"10.1007\/s00779-016-0953-5","article-title":"A framework for physiological indicators of flow in vr games: construction and preliminary evaluation","volume":"20","author":"Bian","year":"2016","journal-title":"Personal Ubiquitous Comput."},{"key":"B5","first-page":"13","article-title":"Boredom, engagement and anxiety as indicators for adaptation to difficulty in games","author":"Chanel","year":"2008"},{"key":"B6","first-page":"44","article-title":"A probabilistic approach for detection and analysis of cognitive flow","author":"Chatterjee","year":"2016"},{"key":"B7","volume-title":"Finding flow: the psychology of engagement with everyday life","author":"Csikszentmihalhi","year":"2020"},{"key":"B8","volume-title":"Beyond boredom and anxiety","author":"Csikszentmihalyi","year":"2000"},{"key":"B9","doi-asserted-by":"publisher","first-page":"815","DOI":"10.1037\/\/0022-3514.56.5.815","article-title":"Optimal experience in work and leisure","volume":"56","author":"Csikszentmihalyi","year":"1989","journal-title":"J. personality Soc. Psychol."},{"key":"B10","doi-asserted-by":"publisher","first-page":"301","DOI":"10.1037\/a0018432","article-title":"The psychophysiology of flow during piano playing","volume":"10","author":"De Manzano","year":"2010","journal-title":"Emotion"},{"key":"B11","doi-asserted-by":"publisher","first-page":"810","DOI":"10.3758\/s13415-018-0606-4","article-title":"Flow experience and the mobilization of attentional resources","volume":"18","author":"de Sampaio Barros","year":"2018","journal-title":"Cognitive, Affect. Behav. Neurosci."},{"key":"B12","first-page":"1","article-title":"Automatic recognition of flow during work activities using context and physiological signals","author":"Di Lascio","year":"2021"},{"key":"B13","doi-asserted-by":"publisher","first-page":"29","DOI":"10.5753\/rbie.2018.26.02.29","article-title":"Flow theory to promote learning in educational systems: is it really relevant?","volume":"26","author":"dos Santos","year":"2018","journal-title":"Rev. Bras. Inform. Educ."},{"key":"B14","doi-asserted-by":"publisher","first-page":"1043","DOI":"10.1161\/01.cir.93.5.1043","article-title":"Heart rate variability: standards of measurement, physiological interpretation, and clinical use","volume":"93","author":"Electrophysiology","year":"1996","journal-title":"Circulation"},{"key":"B15","first-page":"2002","article-title":"Frequency bands effects on qrs detection","volume":"2003","author":"Elgendi","year":"2010","journal-title":"Biosignals"},{"key":"B16","doi-asserted-by":"crossref","first-page":"278","DOI":"10.4324\/9780429468339-17","article-title":"Flow at work: the evolution of a construct","volume-title":"Current issues in work and organizational psychology","author":"Fullagar","year":"2018"},{"key":"B17","doi-asserted-by":"publisher","first-page":"65","DOI":"10.3233\/978-1-61499-282-0-65","article-title":"Psychophysiological correlates of flow during daily activities","volume":"191","author":"Gaggioli","year":"2013","journal-title":"Annu. Rev. Cybertherapy Telemedicine"},{"key":"B18","doi-asserted-by":"publisher","first-page":"36","DOI":"10.1007\/s10846-022-01744-8","article-title":"User experience and physiological response in human-robot collaboration: a preliminary investigation","volume":"106","author":"Gervasi","year":"2022","journal-title":"J. Intelligent Robotic Syst."},{"key":"B19","first-page":"165","article-title":"Using affective trajectories to describe states of flow in interactive art","author":"Gilroy","year":"2009"},{"key":"B20","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1016\/j.ijpsycho.2015.05.001","article-title":"Physiological correlates of the flow experience during computer game playing","volume":"97","author":"Harmat","year":"2015","journal-title":"Int. J. Psychophysiol."},{"key":"B21","first-page":"139","article-title":"Development of nasa-tlx (task load index): results of empirical and theoretical research","volume-title":"Advances in psychology","author":"Hart","year":"1988"},{"key":"B22","unstructured":"Fordigitstress: a multi-modal stress dataset employing a digital job interview scenario\n HeimerlA.\n PrajodP.\n MertesS.\n BaurT.\n KrausM.\n LiuA.\n 2023"},{"key":"B23","doi-asserted-by":"publisher","first-page":"853733","DOI":"10.3389\/fpsyt.2022.853733","article-title":"Getting into the zone: a pilot study of autonomic-cardiac modulation and flow state during piano performance","volume":"13","author":"Jha","year":"2022","journal-title":"Front. Psychiatry"},{"key":"B24","doi-asserted-by":"publisher","first-page":"849","DOI":"10.1016\/j.jesp.2011.02.004","article-title":"Physiological aspects of flow experiences: skills-demand-compatibility effects on heart rate variability and salivary cortisol","volume":"47","author":"Keller","year":"2011","journal-title":"J. Exp. Soc. Psychol."},{"key":"B25","volume-title":"Psychophysiology of flow experience: an explorative study","author":"Kivikangas","year":"2006"},{"key":"B26","doi-asserted-by":"publisher","first-page":"109","DOI":"10.1007\/978-3-319-67431-5_13","article-title":"The psychophysiology of flow: a systematic review of peripheral nervous system features","volume":"2017","author":"Knierim","year":"2018","journal-title":"Inf. Syst. Neurosci. Gmunden Retreat NeuroIS"},{"key":"B27","doi-asserted-by":"crossref","first-page":"239","DOI":"10.1007\/978-3-030-01087-4_29","article-title":"Exploring flow psychophysiology in knowledge work","volume-title":"Information systems and neuroscience: NeuroIS retreat 2018","author":"Knierim","year":"2019"},{"key":"B28","doi-asserted-by":"publisher","first-page":"197","DOI":"10.1002\/hfm.20703","article-title":"Effects of movement speed and predictability in human\u2013robot collaboration","volume":"27","author":"Koppenborg","year":"2017","journal-title":"Hum. Factors Ergonomics Manuf. Serv. Industries"},{"key":"B29","doi-asserted-by":"publisher","first-page":"483","DOI":"10.1515\/auto-2018-0004","article-title":"Impact of trajectory profiles on user stress in close human-robot interaction","volume":"66","author":"K\u00fchnlenz","year":"2018","journal-title":"at-Automatisierungstechnik"},{"key":"B30","doi-asserted-by":"publisher","first-page":"13","DOI":"10.1017\/s0263574706002955","article-title":"Physiological and subjective responses to articulated robot motion","volume":"25","author":"Kuli\u0107","year":"2007","journal-title":"Robotica"},{"key":"B31","unstructured":"Detecting affective flow states of knowledge workers using physiological sensors\n LeeM.\n 2020"},{"key":"B32","first-page":"2980","article-title":"Focal loss for dense object detection","author":"Lin","year":"2017"},{"key":"B33","first-page":"2108","article-title":"Deepflow: detecting optimal user experience from physiological data using deep neural networks","volume-title":"Aamas","author":"Maier","year":"2019"},{"key":"B34","doi-asserted-by":"publisher","first-page":"1689","DOI":"10.3758\/s13428-020-01516-y","article-title":"Neurokit2: a python toolbox for neurophysiological signal processing","volume":"53","author":"Makowski","year":"2021","journal-title":"Behav. Res. methods"},{"key":"B35","doi-asserted-by":"publisher","first-page":"723","DOI":"10.1089\/cyber.2010.0377","article-title":"Why is facebook so successful? psychophysiological measures describe a core flow state while using facebook","volume":"14","author":"Mauri","year":"2011","journal-title":"Cyberpsychology, Behav. Soc. Netw."},{"key":"B36","doi-asserted-by":"publisher","first-page":"18","DOI":"10.1109\/taffc.2017.2740923","article-title":"Affectnet: a database for facial expression, valence, and arousal computing in the wild","volume":"10","author":"Mollahosseini","year":"2019","journal-title":"IEEE Trans. Affect. Comput."},{"key":"B37","doi-asserted-by":"publisher","first-page":"1245857","DOI":"10.3389\/fpsyg.2023.1245857","article-title":"Behavioral patterns in robotic collaborative assembly: comparing neurotypical and autism spectrum disorder participants","volume":"14","author":"Mondellini","year":"2023","journal-title":"Front. Psychol."},{"key":"B38","first-page":"688","article-title":"Stuck and frustrated or in flow and happy: sensing developers\u2019 emotions and progress","author":"M\u00fcller","year":"2015"},{"key":"B39","first-page":"81","article-title":"Flow and immersion in first-person shooters: measuring the player\u2019s gameplay experience","author":"Nacke","year":"2008"},{"key":"B40","doi-asserted-by":"publisher","first-page":"83","DOI":"10.1504\/ijisam.2014.062288","article-title":"Flow in gaming: literature synthesis and framework development","volume":"1","author":"Nah","year":"2014","journal-title":"Int. J. Inf. Syst. Manag."},{"key":"B41","first-page":"105","article-title":"The concept of flow","volume":"89","author":"Nakamura","year":"2002","journal-title":"Handb. Posit. Psychol."},{"key":"B42","first-page":"1","article-title":"Understanding and mapping pleasure, arousal and dominance social signals to robot-avatar behavior","author":"Nunnari","year":"2023"},{"key":"B43","first-page":"2288","article-title":"Engaging the learner: how can the flow experience support e-learning?","volume-title":"E-learn: world conference on E-learning in corporate, government, healthcare, and higher education","author":"Pearce","year":"2005"},{"key":"B44","doi-asserted-by":"publisher","first-page":"139","DOI":"10.1007\/978-1-4614-2359-1_8","article-title":"Psychophysiological correlates of flow-experience","author":"Peifer","year":"2012","journal-title":"Adv. flow Res."},{"key":"B45","doi-asserted-by":"publisher","first-page":"62","DOI":"10.1016\/j.jesp.2014.01.009","article-title":"The relation of flow-experience and physiological arousal under stress\u2014can u shape it?","volume":"53","author":"Peifer","year":"2014","journal-title":"J. Exp. Soc. Psychol."},{"key":"B46","doi-asserted-by":"publisher","first-page":"3998","DOI":"10.3390\/s21123998","article-title":"Heart rate variability in psychology: a review of hrv indices and an analysis tutorial","volume":"21","author":"Pham","year":"2021","journal-title":"Sensors"},{"key":"B47","first-page":"549","article-title":"On the generalizability of ecg-based stress detection models","author":"Prajod","year":"2022"},{"key":"B48","first-page":"311","article-title":"Using explainable ai to identify differences between clinical and experimental pain detection models based on facial expressions","author":"Prajod","year":"2022"},{"key":"B49","first-page":"217","article-title":"Do deep neural networks forget facial action units? exploring the effects of transfer learning in health related facial expression recognition","volume-title":"International workshop on health intelligence","author":"Prajod","year":"2021"},{"key":"B50","unstructured":"Mindbot planetary gearbox\n RedaelliD. F.\n StormF. A.\n FiorettaG.\n 2021"},{"key":"B51","doi-asserted-by":"publisher","first-page":"463","DOI":"10.1109\/taffc.2020.3045269","article-title":"To be or not to be in flow at work: physiological classification of flow using machine learning","volume":"14","author":"Rissler","year":"2020","journal-title":"IEEE Trans. Affect. Comput."},{"key":"B52","doi-asserted-by":"publisher","first-page":"2011","DOI":"10.4236\/psych.2018.98115","article-title":"\u201cbeing in the zone\u201d: a systematic review on the relationship of psychological correlates and the occurrence of flow experiences in sports\u2019 performance","volume":"9","author":"Stamatelopoulou","year":"2018","journal-title":"Psychology"},{"key":"B53","doi-asserted-by":"publisher","first-page":"1121","DOI":"10.3389\/fpsyg.2017.01121","article-title":"Physiological signal analysis for evaluating flow during playing of computer games of varying difficulty","volume":"8","author":"Tian","year":"2017","journal-title":"Front. Psychol."},{"key":"B54","doi-asserted-by":"publisher","first-page":"42","DOI":"10.1038\/s42256-020-00280-0","article-title":"Estimation of continuous valence and arousal levels from faces in naturalistic conditions","volume":"3","author":"Toisoul","year":"2021","journal-title":"Nat. Mach. Intell."},{"key":"B55","doi-asserted-by":"publisher","first-page":"408","DOI":"10.1016\/j.chb.2015.06.023","article-title":"Understanding the psychophysiology of flow: a driving simulator experiment to investigate the relationship between flow and heart rate variability","volume":"52","author":"Tozman","year":"2015","journal-title":"Comput. Hum. Behav."},{"key":"B56","first-page":"831","article-title":"The social signal interpretation (ssi) framework: multimodal signal processing and recognition in real-time","author":"Wagner","year":"2013"},{"key":"B57","doi-asserted-by":"publisher","first-page":"8926","DOI":"10.3390\/s23218926","article-title":"Multimodal assessment of cognitive workload using neural, subjective and behavioural measures in smart factory settings","volume":"23","author":"Zakeri","year":"2023","journal-title":"Sensors"}],"container-title":["Frontiers in Robotics and AI"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/frobt.2024.1393795\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,5,30]],"date-time":"2024-05-30T08:40:42Z","timestamp":1717058442000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/frobt.2024.1393795\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,5,30]]},"references-count":57,"alternative-id":["10.3389\/frobt.2024.1393795"],"URL":"https:\/\/doi.org\/10.3389\/frobt.2024.1393795","relation":{},"ISSN":["2296-9144"],"issn-type":[{"value":"2296-9144","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,5,30]]}}}
