{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2024,7,21]],"date-time":"2024-07-21T18:28:51Z","timestamp":1721586531722},"reference-count":49,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2021,7,29]],"date-time":"2021-07-29T00:00:00Z","timestamp":1627516800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100004359","name":"Vetenskapsr\u00e5det","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100004359","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Robot. AI"],"abstract":"A longstanding barrier to deploying robots in the real world is the ongoing need to author robot behavior. Remote data collection\u2013particularly crowdsourcing\u2014is increasingly receiving interest. In this paper, we make the argument to scale robot programming to the crowd and present an initial investigation of the feasibility of this proposed method. Using an off-the-shelf visual programming interface, non-experts created simple robot programs for two typical robot tasks (navigation and pick-and-place). Each needed four subtasks with an increasing number of programming statements (if statement, while loop, variables) for successful completion of the programs. Initial findings of an online study (N = 279) indicate that non-experts, after minimal instruction, were able to create simple programs using an off-the-shelf visual programming interface. We discuss our findings and identify future avenues for this line of research.<\/jats:p>","DOI":"10.3389\/frobt.2021.646002","type":"journal-article","created":{"date-parts":[[2021,7,29]],"date-time":"2021-07-29T07:41:21Z","timestamp":1627544481000},"update-policy":"http:\/\/dx.doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":8,"title":["Exploring Non-Expert Robot Programming Through Crowdsourcing"],"prefix":"10.3389","volume":"8","author":[{"given":"Sanne","family":"van Waveren","sequence":"first","affiliation":[]},{"given":"Elizabeth J.","family":"Carter","sequence":"additional","affiliation":[]},{"given":"Oscar","family":"\u00d6rnberg","sequence":"additional","affiliation":[]},{"given":"Iolanda","family":"Leite","sequence":"additional","affiliation":[]}],"member":"1965","published-online":{"date-parts":[[2021,7,29]]},"reference":[{"key":"B1","first-page":"5537","article-title":"Roboflow: A Flow-Based Visual Programming Language for mobile Manipulation Tasks","author":"Alexandrova","year":"2015"},{"key":"B2","doi-asserted-by":"publisher","first-page":"469","DOI":"10.1016\/j.robot.2008.10.024","article-title":"A Survey of Robot Learning from Demonstration","volume":"57","author":"Argall","year":"2009","journal-title":"Robotics Autonomous Syst."},{"key":"B3","doi-asserted-by":"crossref","volume-title":"Learnable Programming: Blocks and beyond","author":"Bau","year":"2017","DOI":"10.1145\/3015455"},{"key":"B4","doi-asserted-by":"publisher","first-page":"82","DOI":"10.5898\/jhri.2.1.breazeal","article-title":"Crowdsourcing Human-Robot Interaction: New Methods and System Evaluation in a Public Environment","volume":"2","author":"Breazeal","year":"2013","journal-title":"Jhri"},{"key":"B5","doi-asserted-by":"publisher","first-page":"4","DOI":"10.1201\/9781498710411-35","article-title":"Sus-a Quick and Dirty Usability Scale","volume":"189","author":"Brooke","year":"1996","journal-title":"Usability Eval. Industry"},{"key":"B6","first-page":"14","article-title":"Crowdsourcing Hri through Online Multiplayer Games","author":"Chernova","year":"2010"},{"key":"B7","first-page":"919","article-title":"Iterative Design of a System for Programming Socially Interactive Service Robots","author":"Chung","year":"2016"},{"key":"B8","doi-asserted-by":"publisher","first-page":"100970","DOI":"10.1016\/j.cola.2020.100970","article-title":"Visual Programming Environments for End-User Development of Intelligent and Social Robots, a Systematic Review","volume":"58","author":"Coronado","year":"2020","journal-title":"J. Comp. Languages"},{"key":"B9","doi-asserted-by":"crossref","first-page":"450","DOI":"10.1007\/978-3-319-78963-7_56","article-title":"Design of a Human-Centered Robot Framework for End-User Programming and Applications","volume-title":"ROMANSY 22\u2013Robot Design, Dynamics and Control","author":"Coronado","year":"2019"},{"key":"B10","doi-asserted-by":"crossref","first-page":"62","DOI":"10.1109\/ARSO.2018.8625839","article-title":"Development of Intelligent Behaviors for Social Robots via User-Friendly and Modular Programming Tools","volume-title":"2018 IEEE Workshop on Advanced Robotics and its Social Impacts (ARSO)","author":"Coronado","year":"2018"},{"key":"B11","first-page":"2352","article-title":"Robostudio: A Visual Programming Environment for Rapid Authoring and Customization of Complex Services on a Personal Service Robot","author":"Datta","year":"2012"},{"key":"B12","doi-asserted-by":"crossref","first-page":"1127","DOI":"10.1016\/B978-044481862-1.50113-8","article-title":"End-user Programming","volume-title":"Handbook of Human-Computer Interaction","author":"Eisenberg","year":"1997"},{"key":"B13","first-page":"73","article-title":"Measuring Programming Experience","author":"Feigenspan","year":"2012"},{"key":"B14","first-page":"303","article-title":"Human-robot Interaction Design Using Interaction Composer Eight Years of Lessons Learned","author":"Glas","year":"2016"},{"key":"B15","doi-asserted-by":"crossref","first-page":"89","DOI":"10.7551\/mitpress\/9481.003.0017","article-title":"An Interaction Design Framework for Social Robots","volume-title":"Robotics: Science and Systems","author":"Glas","year":"2012"},{"key":"B16","doi-asserted-by":"publisher","first-page":"131","DOI":"10.1006\/jvlc.1996.0009","article-title":"Usability Analysis of Visual Programming Environments: A \u2018Cognitive Dimensions' Framework","volume":"7","author":"Green","year":"1996","journal-title":"J. Vis. Languages Comput."},{"key":"B17","first-page":"73","article-title":"Conversational Bootstrapping and Other Tricks of a Concierge Robot","author":"Guo","year":"2017"},{"key":"B18","first-page":"139","article-title":"Development of Nasa-Tlx (Task Load index): Results of Empirical and Theoretical Research","volume-title":"Advances in Psychology","author":"Hart","year":"1988"},{"key":"B19","first-page":"453","article-title":"Code3: A System for End-To-End Programming of mobile Manipulator Robots for Novices and Experts","author":"Huang","year":"2017"},{"key":"B20","first-page":"295","article-title":"Design and Evaluation of a Rapid Programming System for Service Robots","author":"Huang","year":"2016"},{"key":"B21","first-page":"1","article-title":"Evorus","author":"Huang","year":"2018"},{"key":"B22","first-page":"14","article-title":"Crowdsourcing a Self-Evolving Dialog Graph","author":"Jonell","year":"2019"},{"key":"B23","first-page":"1","article-title":"Real User Evaluation of Spoken Dialogue Systems Using Amazon Mechanical Turk","author":"Jur\u010d\u00ed\u010dek","year":"2011"},{"key":"B24","first-page":"192","article-title":"Learning and Reusing Dialog for Repeated Interactions with a Situated Social Agent","author":"Kennedy","year":"2017"},{"key":"B25","doi-asserted-by":"publisher","first-page":"e0149787","DOI":"10.1371\/journal.pone.0149787","article-title":"Inter-coder Agreement in One-To-many Classification: Fuzzy Kappa","volume":"11","author":"Kirilenko","year":"2016","journal-title":"PloS one"},{"key":"B26","first-page":"151","article-title":"Chorus: a Crowd-Powered Conversational Assistant","author":"Lasecki","year":"2013"},{"key":"B27","first-page":"109","article-title":"Personifying Programming Tool Feedback Improves Novice Programmers\u2019 Learning","author":"Lee","year":"2011"},{"key":"B28","doi-asserted-by":"publisher","first-page":"291","DOI":"10.1007\/s12369-013-0178-y","article-title":"Social Robots for Long-Term Interaction: a Survey","volume":"5","author":"Leite","year":"2013","journal-title":"Int. J. Soc. Robotics"},{"key":"B29","first-page":"13","article-title":"Semi-situated Learning of Verbal and Nonverbal Content for Repeated Human-Robot Interaction","author":"Leite","year":"2016"},{"key":"B30","doi-asserted-by":"publisher","first-page":"821","DOI":"10.1109\/proc.1983.12681","article-title":"Robot Programming","volume":"71","author":"Lozano-Perez","year":"1983","journal-title":"Proc. IEEE"},{"key":"B31","doi-asserted-by":"publisher","first-page":"223","DOI":"10.1145\/1227504.1227388","article-title":"Scratch for Budding Computer Scientists","volume":"39","author":"Malan","year":"2007","journal-title":"SIGCSE Bull."},{"key":"B32","doi-asserted-by":"publisher","first-page":"172","DOI":"10.3115\/v1\/w14-5003","article-title":"Crowdsourcing Language Generation Templates for Dialogue Systems","author":"Mitchell","year":"2014"},{"key":"B33","first-page":"385","article-title":"Automatic Learning and Generation of Social Behavior from Collective Human Gameplay","author":"Orkin","year":"2009"},{"key":"B34","first-page":"39","article-title":"The Restaurant Game: Learning Social Behavior and Language from Thousands of Players Online","volume":"3","author":"Orkin","year":"2007","journal-title":"J. Game Dev."},{"key":"B35","first-page":"447","article-title":"Computer Science Outreach with End-User Robot-Programming Tools","author":"Paramasivam","year":"2017"},{"key":"B36","first-page":"75","article-title":"Authoring and Verifying Human-Robot Interactions","author":"Porfirio","year":"2018"},{"key":"B37","first-page":"46","article-title":"Choregraphe: a Graphical Tool for Humanoid Robot Programming","author":"Pot","year":"2009"},{"key":"B38","doi-asserted-by":"publisher","first-page":"60","DOI":"10.1145\/1592761.1592779","article-title":"Scratch","volume":"52","author":"Resnick","year":"2009","journal-title":"Commun. Acm"},{"key":"B39","doi-asserted-by":"publisher","first-page":"301","DOI":"10.1016\/j.ijhcs.2011.11.004","article-title":"A Crowdsourcing Method to Develop Virtual Human Conversational Agents","volume":"70","author":"Rossen","year":"2012","journal-title":"Int. J. Human-Computer Stud."},{"key":"B40","first-page":"1439","article-title":"Design Patterns for Exploring and Prototyping Human-Robot Interactions","author":"Saupp\u00e9","year":"2014"},{"key":"B41","first-page":"473","article-title":"Situated Tangible Robot Programming","author":"Sefidgar","year":"2017"},{"key":"B42","first-page":"319","article-title":"End-user Programming of Manipulator Robots in Situated Tangible Programming Paradigm","author":"Sefidgar","year":"2018"},{"key":"B43","first-page":"141","article-title":"Robotist: Interactive Situated Tangible Robot Programming","author":"Sefidgar","year":"2018"},{"key":"B44","first-page":"463","article-title":"Simplified Programming of Re-usable Skills on a Safe Industrial Robot: Prototype and Evaluation","author":"Stenmark","year":"2017"},{"key":"B45","doi-asserted-by":"publisher","first-page":"2","DOI":"10.1145\/3310356","article-title":"Blossom: A Handcrafted Open-Source Robot","volume":"8","author":"Suguitan","year":"2019","journal-title":"ACM Trans. Human-Robot Interaction (Thri)"},{"key":"B46","volume-title":"Robobarista: Learning to Manipulate Novel Objects via Deep Multimodal Embedding","author":"Sung","year":"2016"},{"key":"B47","first-page":"5","article-title":"Blockly Language Creation and Applications: Visual Programming for media Computation and Bluetooth Robotics Control","author":"Trower","year":"2015"},{"key":"B48","volume-title":"Robot Accident Investigation: A Case Study in Responsible Robotics","author":"Winfield","year":"2020"},{"key":"B49","first-page":"102","article-title":"Chatbot Evaluation and Database Expansion via Crowdsourcing","author":"Yu","year":"2016"}],"container-title":["Frontiers in Robotics and AI"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/frobt.2021.646002\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,11,6]],"date-time":"2023-11-06T16:34:06Z","timestamp":1699288446000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/frobt.2021.646002\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,7,29]]},"references-count":49,"alternative-id":["10.3389\/frobt.2021.646002"],"URL":"https:\/\/doi.org\/10.3389\/frobt.2021.646002","relation":{},"ISSN":["2296-9144"],"issn-type":[{"value":"2296-9144","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,7,29]]}}}
