Abstract
To achieve psychological inclusion and skill development orientation in human skill training, this paper proposes a haptic-guided training strategy generation method with Deep Reinforcement Learning (DRL)-based agent as the core and Zone of Proximal Development (ZPD) tuning as the auxiliary. The information of the expert and trainee is stored first with a designed database that can be accessed in real-time, which establishes the data foundation. Then, under the DRL framework, a strategy generation agent is designed, which consists of an actor-network and two Q-networks. The former network generates the agent’s decision policy, while the other two Q-networks work to approximate the state-action value function, and the parameters of all of them are administrated by the Soft Actor-Critic (SAC) algorithm. In addition, for the first time, the psychological ZPD evaluation method is integrated into the strategy generation of the DRL-based agent, which is utilized to describe the relationship between a trainees intrinsic skills and guidance. With it, the problem of transitional guidance or insufficient guidance can be handled well. Finally, simulation experiments validate the proposed method, demonstrating its efficiency in regulating the trainee under favorable training conditions.
Article PDF
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Avoid common mistakes on your manuscript.
Data Availability
The Code and data are available.
Code Availability
The code that support the fndings of this study is available from the corresponding author, [author initials], upon reasonable request. No data were used.
References
Xie, B., Liu, H., Alghofaili, R., et al.: A review on virtual reality skill training applications. Front. Virtual. Real. 2021(2), 645153 (2021)
Zhang, Q., Li, B.: Relative hidden markov models for video-based evaluation of motion skills in surgical training. IEEE Trans. Pattern Anal. Mach. Intell. 37(6), 1206–1218 (2014)
Ershad, M., Rege, R., Fey, A.: Adaptive surgical robotic training using real-time stylistic behavior feedback through haptic cues. IEEE Trans. Med. Robot. Bionics. 3(4), 959–969 (2021)
Wulf, G., Shea, C., Lewthwaite, R.: Motor skill learning and performance: a review of influential factors. Med. Educ. 44(1), 75–84 (2010)
Caccianiga, G., Mariani, A., de Paratesi, C., et al.: Multi-sensory guidance and feedback for simulation-based training in robot assisted surgery: a preliminary comparison of visual, haptic, and visuo-haptic. IEEE Robot. Autom. Lett. 6(2), 3801–3808 (2021)
Darvish, K., Penco, L., Ramos, J., et al.: Teleoperation of humanoid robots: a survey. IEEE Trans. Robot. 39(3), 1706–1727 (2023)
Shahbazi, M., Atashzar, S., Ward, C., et al.: Multimodal sensorimotor integration for expert-in-the-loop telerobotic surgical training. IEEE Trans. Robot. 34(6), 1549–1564 (2018)
Chi, W., Rafii-Tari, H., Payne, C., et al.: A learning based training and skill assessment platform with haptic guidance for endovascular catheterization. IEEE International Conference on Robotics and Automation (ICRA), 2357-2363 (2017)
Zhang, Y., Li, S., Nolan, K. et al.: Adaptive assist-as-needed control based on actor-critic reinforcement learning. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 4066-4071 (2019)
Liu, G., Lu, K., Zhang, Y.: Haptic-based training for tank gunnery using decoupled motion control. IEEE Comput. Graph. Appl. 33(2), 73–79 (2013)
Liu, G., Lu, K., Zhang, Y.: Networked haptic interaction to implement hand in “ hand’’ human motor skill training for tank gunnery. Int. J. Adv. Robot. Syst. 10(135), 1–12 (2013)
Park, W., Babushkin, V., Tahir, S., et al.: Haptic guidance to support handwriting for children with cognitive and fine motor delays. IEEE Trans. Haptics 14(3), 626–634 (2021)
Paez Granados, D., Yamamoto, B., Kamide, H., et al.: Dance teaching by a robot: combining cognitive and physical human-robot interaction for supporting the skill learning process. IEEE Robot. Autom. Lett. 2(3), 1452–1459 (2017)
Paez Granados, D., Kinugawa, J., Hirata, Y., et al.: Guiding human motions in physical human-robot interaction through com motion control of a dance teaching robot. IEEE-RAS Int. Conf. Humanoid Robots 279-285 (2017)
Hirokawa, M., Uesugi, N., Furugori, S., et al.: A haptic instruction based assisted driving system for training the reverse parking. IEEE Int. Conf. Robot. Autom. 3713-3718 (2012)
Mariani, A., Pellegrini, E., De Momi, E.: Skill-oriented and performance-driven adaptive curricula for training in robot-assisted surgery using simulators: a feasibility study. IEEE Trans. Biomed. Eng. 68(2), 685–694 (2021)
Smith, C., Pezent, E., O’Malley, M.: Spatially separated cutaneous haptic guidance for training of a virtual sensorimotor task. IEEE Haptics Symposium (HAPTICS), 974-979 (2020)
Liu, L., Liu, G., Zhang, Y.: A novel haptic training method through skill decomposition. World Haptics Conference, 621-625 (2013)
Gibo, T., Abbink, D.: Movement strategy discovery during training via haptic guidance. IEEE Trans. Haptics 9(2), 243–254 (2016)
Hara, T., Sato, T., Ogata, T., et al.: Uncertainty-aware haptic shared control with humanoid robots for flexible object manipulation. IEEE Robot. Autom. Lett. 8(10), 6435–6442 (2023)
Tong, Y., Liu, H., Zhang, Z.: Advancements in humanoid robots: a comprehensive review and future prospects. IEEE/CAA J. Autom. Sin. 11(2), 301–328 (2024)
Rowland, D., Davis, B., Higgins, T., et al.: Enhancing user performance by adaptively changing haptic feedback cues in a fitts’s law task. IEEE Transactions on Haptics (Early Access), (2024)
Huang, X., Wang, X., Zhao, Y., et al.: Guided model-based policy search method for aast motor learning of robots with learned dynamics. IEEE Trans. Autom. Sci. Eng. (Early Acess) (2024). https://doi.org/10.1109/TASE.2024.3352580
Qu, M., Wang, Y., Pham, D.: Robotic disassembly task training and skill transfer using reinforcement learning. IEEE Trans. Ind. Inform. 19(11), 10934–10943 (2023)
Dewa, C., Miura, J.: Integrating multiple policies for person-following robot training using deep reinforcement learning. IEEE Access 2021(9), 75526–75541 (2021)
Tian, X., Pan, B., Bai, L., et al.: Fruit picking robot arm training solution based on reinforcement learning in digital twin. J. ICT Stand. 11(3), 261–282 (2023)
Guzman, L., Morellas, V., Papanikolopoulos, N.: Robotic embodiment of human-like motor skills via reinforcement learning. IEEE Robot Autom Lett 7(2), 3711–3717 (2022)
Xiang, G., Su, J.: Task-oriented deep reinforcement learning for robotic skill acquisition and control. EEE Trans. Cybern. 51(2), 1056–1069 (2021)
Jiang, L., Wang, Y.: A personalized computational model for human-like automated decision-making. IEEE Trans. Autom. Sci. Eng. 19(2), 850–863 (2022)
Wiltshire, T., Fiore, S.: Social cognitive and affective neuroscience in human-machine systems: a roadmap for improving training, human-robot interaction, and team performance. IEEE Trans. Hum.-Mach. Syst. 44(6), 779–787 (2014)
Mabry, B.: The zone of proximal development (ZPD): the power of just right. https://www.nwea.org/blog/2020/the-zone-of-proximal-development-zpd-the-power-of-just-right/ [Online;] (2020)
Zhang, S., Lai, W., Song, J., et al.: Scaffolding instruction design research based on zone of proximal development of learning community. International Conference of Educational Innovation Through Technology, 258-262 (2018)
Puzi, A., Sidek, S., Sado, F.: Mechanical impedance modeling of human arm: a survey. IOP Conf. Ser. Mater. Sci. Eng. 184(1), 012041 (2017) IOP Publishing
Khalil, H.: Nonlinear systems third edition. Upper Saddle River Nj Prentice Hall Inc, 262-266 (2002)
Haarnoja, T., Zhou, A., Abbeel, P., et al.: Soft actor-critic: off-policy maximum entropy deep reinforcement learning with a stochastic actor. International Conference on Machine Learning, 1861-1870 (2018)
Haarnoja, T., Zhou, A., Hartikainen, K., et al.: Soft actor-critic algorithms and applications. arXiv:1812.05905 (2018)
Hida, N., Abid, M., Lakrad, F.: A nonlinear model of the hand-arm system and parameters identification using vibration transmissibility. EDP Sci. 2018(241), 01014 (2018)
Fu, M., Cavusoglu, M.: Human-arm-and-hand-dynamic model with variability analyses for a stylus-based haptic interface. IEEE Trans. Syst. Man. Cybern. B Cybern. 42(6)£\(^{\rm o}\)1633-1644 (2012)
Acknowledgements
This work was supported in part by the National Natural Science Foundation of China under Grant 62273280, 62303493, 62103334, and 92370123. This research has no Conflicts of interest/Competing interests.
Funding
This research is sponsored by the National Natural Science Foundation of China (Grant No: 62273280, 62303493, 62103334, 92370123).
Author information
Authors and Affiliations
Contributions
In this article, Yang Yang completed the problem research and formulation, designed the methodology, implementation of the code, and completed the experiment and data analysis. Completed the writing of the paper. This research will deepest gratitude to Prof. Panfeng Huang, Yang’s supervisor, for his good platform and resource support for the research. Second, it would like to express the heartfelt gratitude to Prof. Xing Liu for his constant encouragement and research guidance. Lastly, Prof. Haifei Chen provided many suggestions and advice on writing and research methods. The article was published with the consent of all authors.
Corresponding author
Ethics declarations
We declare that we have no financial and personal relationships with other people or organizations that can inappropriately influence our work, there is no professional or other personal interest of any nature or kind in any product, service and/or company that could be constructed as influencing the position presented in, or the review of, the manuscript entitled.
Ethics Approval:
Not Applicable.
Consent to Participate
Not Applicable.
Consent to Publish
Not Applicable.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Yang, Y., Chen, H., Liu, X. et al. Guidance-As-Progressive in Human Skill Training Based on Deep Reinforcement Learning. J Intell Robot Syst 110, 116 (2024). https://doi.org/10.1007/s10846-024-02147-7
Received:
Accepted:
Published:
DOI: https://doi.org/10.1007/s10846-024-02147-7