{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2024,9,27]],"date-time":"2024-09-27T04:26:56Z","timestamp":1727411216129},"reference-count":38,"publisher":"Springer Science and Business Media LLC","issue":"3","license":[{"start":{"date-parts":[[2024,8,3]],"date-time":"2024-08-03T00:00:00Z","timestamp":1722643200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2024,8,3]],"date-time":"2024-08-03T00:00:00Z","timestamp":1722643200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["62273280","62103334","92370123"],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["J Intell Robot Syst"],"abstract":"Abstract<\/jats:title>To achieve psychological inclusion and skill development orientation in human skill training, this paper proposes a haptic-guided training strategy generation method with Deep Reinforcement Learning (DRL)-based agent as the core and Zone of Proximal Development (ZPD) tuning as the auxiliary. The information of the expert and trainee is stored first with a designed database that can be accessed in real-time, which establishes the data foundation. Then, under the DRL framework, a strategy generation agent is designed, which consists of an actor-network and two Q-networks. The former network generates the agent\u2019s decision policy, while the other two Q-networks work to approximate the state-action value function, and the parameters of all of them are administrated by the Soft Actor-Critic (SAC) algorithm. In addition, for the first time, the psychological ZPD evaluation method is integrated into the strategy generation of the DRL-based agent, which is utilized to describe the relationship between a trainees intrinsic skills and guidance. With it, the problem of transitional guidance or insufficient guidance can be handled well. Finally, simulation experiments validate the proposed method, demonstrating its efficiency in regulating the trainee under favorable training conditions.<\/jats:p>","DOI":"10.1007\/s10846-024-02147-7","type":"journal-article","created":{"date-parts":[[2024,8,3]],"date-time":"2024-08-03T08:01:53Z","timestamp":1722672113000},"update-policy":"http:\/\/dx.doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Guidance-As-Progressive in Human Skill Training Based on Deep Reinforcement Learning"],"prefix":"10.1007","volume":"110","author":[{"given":"Yang","family":"Yang","sequence":"first","affiliation":[]},{"given":"Haifei","family":"Chen","sequence":"additional","affiliation":[]},{"given":"Xing","family":"Liu","sequence":"additional","affiliation":[]},{"ORCID":"http:\/\/orcid.org\/0000-0002-5132-9602","authenticated-orcid":false,"given":"Panfeng","family":"Huang","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2024,8,3]]},"reference":[{"issue":"2","key":"2147_CR1","doi-asserted-by":"publisher","first-page":"645153","DOI":"10.3389\/frvir.2021.645153","volume":"2021","author":"B Xie","year":"2021","unstructured":"Xie, B., Liu, H., Alghofaili, R., et al.: A review on virtual reality skill training applications. Front. Virtual. Real. 2021(2), 645153 (2021)","journal-title":"Front. Virtual. Real."},{"issue":"6","key":"2147_CR2","doi-asserted-by":"publisher","first-page":"1206","DOI":"10.1109\/TPAMI.2014.2361121","volume":"37","author":"Q Zhang","year":"2014","unstructured":"Zhang, Q., Li, B.: Relative hidden markov models for video-based evaluation of motion skills in surgical training. IEEE Trans. Pattern Anal. Mach. Intell. 37(6), 1206\u20131218 (2014)","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"issue":"4","key":"2147_CR3","doi-asserted-by":"publisher","first-page":"959","DOI":"10.1109\/TMRB.2021.3124128","volume":"3","author":"M Ershad","year":"2021","unstructured":"Ershad, M., Rege, R., Fey, A.: Adaptive surgical robotic training using real-time stylistic behavior feedback through haptic cues. IEEE Trans. Med. Robot. Bionics. 3(4), 959\u2013969 (2021)","journal-title":"IEEE Trans. Med. Robot. Bionics."},{"issue":"1","key":"2147_CR4","doi-asserted-by":"publisher","first-page":"75","DOI":"10.1111\/j.1365-2923.2009.03421.x","volume":"44","author":"G Wulf","year":"2010","unstructured":"Wulf, G., Shea, C., Lewthwaite, R.: Motor skill learning and performance: a review of influential factors. Med. Educ. 44(1), 75\u201384 (2010)","journal-title":"Med. Educ."},{"doi-asserted-by":"crossref","unstructured":"Caccianiga, G., Mariani, A., de Paratesi, C., et al.: Multi-sensory guidance and feedback for simulation-based training in robot assisted surgery: a preliminary comparison of visual, haptic, and visuo-haptic. IEEE Robot. Autom. Lett. 6(2), 3801\u20133808 (2021)","key":"2147_CR5","DOI":"10.1109\/LRA.2021.3063967"},{"doi-asserted-by":"crossref","unstructured":"Darvish, K., Penco, L., Ramos, J., et al.: Teleoperation of humanoid robots: a survey. IEEE Trans. Robot. 39(3), 1706\u20131727 (2023)","key":"2147_CR6","DOI":"10.1109\/TRO.2023.3236952"},{"issue":"6","key":"2147_CR7","doi-asserted-by":"publisher","first-page":"1549","DOI":"10.1109\/TRO.2018.2861916","volume":"34","author":"M Shahbazi","year":"2018","unstructured":"Shahbazi, M., Atashzar, S., Ward, C., et al.: Multimodal sensorimotor integration for expert-in-the-loop telerobotic surgical training. IEEE Trans. Robot. 34(6), 1549\u20131564 (2018)","journal-title":"IEEE Trans. Robot."},{"doi-asserted-by":"crossref","unstructured":"Chi, W., Rafii-Tari, H., Payne, C., et al.: A learning based training and skill assessment platform with haptic guidance for endovascular catheterization. IEEE International Conference on Robotics and Automation (ICRA), 2357-2363 (2017)","key":"2147_CR8","DOI":"10.1109\/ICRA.2017.7989273"},{"doi-asserted-by":"crossref","unstructured":"Zhang, Y., Li, S., Nolan, K. et al.: Adaptive assist-as-needed control based on actor-critic reinforcement learning. IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS), 4066-4071 (2019)","key":"2147_CR9","DOI":"10.1109\/IROS40897.2019.8968464"},{"issue":"2","key":"2147_CR10","doi-asserted-by":"publisher","first-page":"73","DOI":"10.1109\/MCG.2013.21","volume":"33","author":"G Liu","year":"2013","unstructured":"Liu, G., Lu, K., Zhang, Y.: Haptic-based training for tank gunnery using decoupled motion control. IEEE Comput. Graph. Appl. 33(2), 73\u201379 (2013)","journal-title":"IEEE Comput. Graph. Appl."},{"issue":"135","key":"2147_CR11","first-page":"1","volume":"10","author":"G Liu","year":"2013","unstructured":"Liu, G., Lu, K., Zhang, Y.: Networked haptic interaction to implement hand in \u201c hand\u2019\u2019 human motor skill training for tank gunnery. Int. J. Adv. Robot. Syst. 10(135), 1\u201312 (2013)","journal-title":"Int. J. Adv. Robot. Syst."},{"issue":"3","key":"2147_CR12","doi-asserted-by":"publisher","first-page":"626","DOI":"10.1109\/TOH.2021.3068786","volume":"14","author":"W Park","year":"2021","unstructured":"Park, W., Babushkin, V., Tahir, S., et al.: Haptic guidance to support handwriting for children with cognitive and fine motor delays. IEEE Trans. Haptics 14(3), 626\u2013634 (2021)","journal-title":"IEEE Trans. Haptics"},{"doi-asserted-by":"crossref","unstructured":"Paez Granados, D., Yamamoto, B., Kamide, H., et al.: Dance teaching by a robot: combining cognitive and physical human-robot interaction for supporting the skill learning process. IEEE Robot. Autom. Lett. 2(3), 1452\u20131459 (2017)","key":"2147_CR13","DOI":"10.1109\/LRA.2017.2671428"},{"doi-asserted-by":"crossref","unstructured":"Paez Granados, D., Kinugawa, J., Hirata, Y., et al.: Guiding human motions in physical human-robot interaction through com motion control of a dance teaching robot. IEEE-RAS Int. Conf. Humanoid Robots 279-285 (2017)","key":"2147_CR14","DOI":"10.1109\/HUMANOIDS.2016.7803289"},{"doi-asserted-by":"crossref","unstructured":"Hirokawa, M., Uesugi, N., Furugori, S., et al.: A haptic instruction based assisted driving system for training the reverse parking. IEEE Int. Conf. Robot. Autom. 3713-3718 (2012)","key":"2147_CR15","DOI":"10.1109\/ICRA.2012.6225193"},{"doi-asserted-by":"crossref","unstructured":"Mariani, A., Pellegrini, E., De Momi, E.: Skill-oriented and performance-driven adaptive curricula for training in robot-assisted surgery using simulators: a feasibility study. IEEE Trans. Biomed. Eng. 68(2), 685\u2013694 (2021)","key":"2147_CR16","DOI":"10.1109\/TBME.2020.3011867"},{"doi-asserted-by":"crossref","unstructured":"Smith, C., Pezent, E., O\u2019Malley, M.: Spatially separated cutaneous haptic guidance for training of a virtual sensorimotor task. IEEE Haptics Symposium (HAPTICS), 974-979 (2020)","key":"2147_CR17","DOI":"10.1109\/HAPTICS45997.2020.ras.HAP20.11.2032900c"},{"doi-asserted-by":"crossref","unstructured":"Liu, L., Liu, G., Zhang, Y.: A novel haptic training method through skill decomposition. World Haptics Conference, 621-625 (2013)","key":"2147_CR18","DOI":"10.1109\/WHC.2013.6548480"},{"issue":"2","key":"2147_CR19","doi-asserted-by":"publisher","first-page":"243","DOI":"10.1109\/TOH.2016.2516984","volume":"9","author":"T Gibo","year":"2016","unstructured":"Gibo, T., Abbink, D.: Movement strategy discovery during training via haptic guidance. IEEE Trans. Haptics 9(2), 243\u2013254 (2016)","journal-title":"IEEE Trans. Haptics"},{"issue":"10","key":"2147_CR20","doi-asserted-by":"publisher","first-page":"6435","DOI":"10.1109\/LRA.2023.3306668","volume":"8","author":"T Hara","year":"2023","unstructured":"Hara, T., Sato, T., Ogata, T., et al.: Uncertainty-aware haptic shared control with humanoid robots for flexible object manipulation. IEEE Robot. Autom. Lett. 8(10), 6435\u20136442 (2023)","journal-title":"IEEE Robot. Autom. Lett."},{"doi-asserted-by":"crossref","unstructured":"Tong, Y., Liu, H., Zhang, Z.: Advancements in humanoid robots: a comprehensive review and future prospects. IEEE\/CAA J. Autom. Sin. 11(2), 301\u2013328 (2024)","key":"2147_CR21","DOI":"10.1109\/JAS.2023.124140"},{"doi-asserted-by":"crossref","unstructured":"Rowland, D., Davis, B., Higgins, T., et al.: Enhancing user performance by adaptively changing haptic feedback cues in a fitts\u2019s law task. IEEE Transactions on Haptics (Early Access), (2024)","key":"2147_CR22","DOI":"10.1109\/TOH.2024.3358188"},{"key":"2147_CR23","doi-asserted-by":"publisher","DOI":"10.1109\/TASE.2024.3352580","author":"X Huang","year":"2024","unstructured":"Huang, X., Wang, X., Zhao, Y., et al.: Guided model-based policy search method for aast motor learning of robots with learned dynamics. IEEE Trans. Autom. Sci. Eng. (Early Acess) (2024). https:\/\/doi.org\/10.1109\/TASE.2024.3352580","journal-title":"IEEE Trans. Autom. Sci. Eng. (Early Acess)"},{"issue":"11","key":"2147_CR24","doi-asserted-by":"publisher","first-page":"10934","DOI":"10.1109\/TII.2023.3242831","volume":"19","author":"M Qu","year":"2023","unstructured":"Qu, M., Wang, Y., Pham, D.: Robotic disassembly task training and skill transfer using reinforcement learning. IEEE Trans. Ind. Inform. 19(11), 10934\u201310943 (2023)","journal-title":"IEEE Trans. Ind. Inform."},{"issue":"9","key":"2147_CR25","doi-asserted-by":"publisher","first-page":"75526","DOI":"10.1109\/ACCESS.2021.3082136","volume":"2021","author":"C Dewa","year":"2021","unstructured":"Dewa, C., Miura, J.: Integrating multiple policies for person-following robot training using deep reinforcement learning. IEEE Access 2021(9), 75526\u201375541 (2021)","journal-title":"IEEE Access"},{"issue":"3","key":"2147_CR26","first-page":"261","volume":"11","author":"X Tian","year":"2023","unstructured":"Tian, X., Pan, B., Bai, L., et al.: Fruit picking robot arm training solution based on reinforcement learning in digital twin. J. ICT Stand. 11(3), 261\u2013282 (2023)","journal-title":"J. ICT Stand."},{"doi-asserted-by":"crossref","unstructured":"Guzman, L., Morellas, V., Papanikolopoulos, N.: Robotic embodiment of human-like motor skills via reinforcement learning. IEEE Robot Autom Lett 7(2), 3711\u20133717 (2022)","key":"2147_CR27","DOI":"10.1109\/LRA.2022.3147453"},{"issue":"2","key":"2147_CR28","doi-asserted-by":"publisher","first-page":"1056","DOI":"10.1109\/TCYB.2019.2949596","volume":"51","author":"G Xiang","year":"2021","unstructured":"Xiang, G., Su, J.: Task-oriented deep reinforcement learning for robotic skill acquisition and control. EEE Trans. Cybern. 51(2), 1056\u20131069 (2021)","journal-title":"EEE Trans. Cybern."},{"issue":"2","key":"2147_CR29","doi-asserted-by":"publisher","first-page":"850","DOI":"10.1109\/TASE.2021.3060727","volume":"19","author":"L Jiang","year":"2022","unstructured":"Jiang, L., Wang, Y.: A personalized computational model for human-like automated decision-making. IEEE Trans. Autom. Sci. Eng. 19(2), 850\u2013863 (2022)","journal-title":"IEEE Trans. Autom. Sci. Eng."},{"doi-asserted-by":"crossref","unstructured":"Wiltshire, T., Fiore, S.: Social cognitive and affective neuroscience in human-machine systems: a roadmap for improving training, human-robot interaction, and team performance. IEEE Trans. Hum.-Mach. Syst. 44(6), 779\u2013787 (2014)","key":"2147_CR30","DOI":"10.1109\/THMS.2014.2343996"},{"unstructured":"Mabry, B.: The zone of proximal development (ZPD): the power of just right. https:\/\/www.nwea.org\/blog\/2020\/the-zone-of-proximal-development-zpd-the-power-of-just-right\/ [Online;] (2020)","key":"2147_CR31"},{"doi-asserted-by":"crossref","unstructured":"Zhang, S., Lai, W., Song, J., et al.: Scaffolding instruction design research based on zone of proximal development of learning community. International Conference of Educational Innovation Through Technology, 258-262 (2018)","key":"2147_CR32","DOI":"10.1109\/EITT.2018.00061"},{"doi-asserted-by":"crossref","unstructured":"Puzi, A., Sidek, S., Sado, F.: Mechanical impedance modeling of human arm: a survey. IOP Conf. Ser. Mater. Sci. Eng. 184(1), 012041 (2017) IOP Publishing","key":"2147_CR33","DOI":"10.1088\/1757-899X\/184\/1\/012041"},{"unstructured":"Khalil, H.: Nonlinear systems third edition. Upper Saddle River Nj Prentice Hall Inc, 262-266 (2002)","key":"2147_CR34"},{"unstructured":"Haarnoja, T., Zhou, A., Abbeel, P., et al.: Soft actor-critic: off-policy maximum entropy deep reinforcement learning with a stochastic actor. International Conference on Machine Learning, 1861-1870 (2018)","key":"2147_CR35"},{"unstructured":"Haarnoja, T., Zhou, A., Hartikainen, K., et al.: Soft actor-critic algorithms and applications. arXiv:1812.05905 (2018)","key":"2147_CR36"},{"issue":"241","key":"2147_CR37","first-page":"01014","volume":"2018","author":"N Hida","year":"2018","unstructured":"Hida, N., Abid, M., Lakrad, F.: A nonlinear model of the hand-arm system and parameters identification using vibration transmissibility. EDP Sci. 2018(241), 01014 (2018)","journal-title":"EDP Sci."},{"doi-asserted-by":"crossref","unstructured":"Fu, M., Cavusoglu, M.: Human-arm-and-hand-dynamic model with variability analyses for a stylus-based haptic interface. IEEE Trans. Syst. Man. Cybern. B Cybern. 42(6)\u00a3$$^{\\rm o}$$1633-1644 (2012)","key":"2147_CR38","DOI":"10.1109\/TSMCB.2012.2197387"}],"container-title":["Journal of Intelligent & Robotic Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10846-024-02147-7.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10846-024-02147-7\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10846-024-02147-7.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,9,26]],"date-time":"2024-09-26T13:14:55Z","timestamp":1727356495000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10846-024-02147-7"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,8,3]]},"references-count":38,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2024,9]]}},"alternative-id":["2147"],"URL":"https:\/\/doi.org\/10.1007\/s10846-024-02147-7","relation":{},"ISSN":["1573-0409"],"issn-type":[{"type":"electronic","value":"1573-0409"}],"subject":[],"published":{"date-parts":[[2024,8,3]]},"assertion":[{"value":"11 October 2023","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"29 June 2024","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"3 August 2024","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"We declare that we have no financial and personal relationships with other people or organizations that can inappropriately influence our work, there is no professional or other personal interest of any nature or kind in any product, service and\/or company that could be constructed as influencing the position presented in, or the review of, the manuscript entitled.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declaration"}},{"value":"Not Applicable.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics Approval:"}},{"value":"Not Applicable.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent to Participate"}},{"value":"Not Applicable.","order":4,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent to Publish"}}],"article-number":"116"}}
