Supervised Classification of Motor-Rehabilitation Body Movements with RGB Cameras and Pose Tracking Data

Luis Guilherme Silva Rodrigues; Diego Roberto Colombo Dias; Marcelo de Paiva Guimarães; Alexandre Fonseca Brandão; Leonardo C. Rocha; Rogério Luiz Iope; José Remo Ferreira Brega

doi:10.5753/jis.2022.2409

Authors

Luis Guilherme Silva Rodrigues Universidade Estadual Paulista https://orcid.org/0000-0001-8729-116X
Diego Roberto Colombo Dias Universidade Federal de São João del-Rei https://orcid.org/0000-0001-9619-2171
Marcelo de Paiva Guimarães Universidade Federal de São Paulo https://orcid.org/0000-0001-5026-1581
Alexandre Fonseca Brandão Universidade Estadual de Campinas https://orcid.org/0000-0003-3978-8862
Leonardo C. Rocha Universidade Federal de São João del-Rei https://orcid.org/0000-0002-4913-4902
Rogério Luiz Iope Universidade Estadual Paulista https://orcid.org/0000-0002-9462-3350
José Remo Ferreira Brega Universidade Estadual Paulista https://orcid.org/0000-0002-2275-4722

DOI:

https://doi.org/10.5753/jis.2022.2409

Keywords:

Classification, Computer Vision, Machine Learning, Pose Tracking, Supervised Learning, Motor-rehabilitation

Abstract

The technological evolution allowed the use of a single camera for precise and effective body tracking, reducing the cost and increasing the accessibility of applications in places where depth cameras and wearable sensors are not available. This paper describes and implements a supervised machine learning process consisting of a mobile application used as a motion capture device which also transforms the data into an input for a machine learning model that classifies upper and lower limbs movements (24 types of human movements). The user performs movements in front of the camera, and the trained model classifies them. We designed the system to work in a motor-rehabilitation context to assist the professional while the patient does physical exercises. The implementation can summarize the movements made during the rehabilitation sessions by counting the repetitions and classifying them when done completely or reached a specific range of motion.

Downloads

Download data is not yet available.

References

Aung, Y. M. and Al-Jumaily, A. (2012). Shoulder rehabilitation with biofeedback simulation. In 2012 IEEE International Conference on Mechatronics and Automation, pages 974–979, Chengdu, China. IEEE.

Bazarevsky, V., Grishchenko, I., Raveendran, K., Zhu, T., Zhang, F., and Grundmann, M. (2020). Blazepose: On-device real-time body pose tracking. In CVPR Workshop on Computer Vision for Augmented and Virtual Reality, Seattle, WA, USA. ArXiv.

Bloom, V., Argyriou, V., and Makris, D. (2016). Hierarchical transfer learning for online recognition of compound actions. Computer Vision and Image Understanding, 144(C):62–72.

Borich, M. R., Wolf, S. L., Tan, A. Q., and Palmer, J. A. (2018). Targeted Neuromodulation of Abnormal Interhemispheric Connectivity to Promote Neural Plasticity and Recovery of Arm Function after Stroke: A Randomized Crossover Clinical Trial Study Protocol. Neural Plasticity, 2018.

Breedon, P., Byrom, B., Siena, L., and Muehlhausen, W. (2016). Enhancing the measurement of clinical outcomes using microsoft kinect. In 2016 International Conference on Interactive Technologies and Games (ITAG), pages 61–69, Nottingham, UK. IEEE.

Choubik, Y. and Mahmoudi, A. (2016). Machine learning for real time poses classification using kinect skeleton data. In 2016 13th International Conference on Computer Graphics, Imaging and Visualization (CGiV), pages 307–311, Beni Mellal, Morocco. IEEE.

Dimyan, M. A. and Cohen, L. G. (2011). Neuroplasticity in the context of motor rehabilitation after stroke. Nature Reviews Neurology, 7(2):76–85.

Dinevan, A., Aung, Y. M., and Al-Jumaily, A. (2011). Human computer interactive system for fast recovery based stroke rehabilitation. In 2011 11th International Conference on Hybrid Intelligent Systems (HIS), pages 647–652, Melacca, Malaysia. IEEE.

Farooq, A. and Won, C. S. (2015). A survey of human action recognition approaches that use an rgb-d sensor. IEIE Transactions on Smart Processing and Computing, 4(4):281–290.

Ijjina, E. P. and Mohan, C. K. (2014). Human action recognition based on mocap information using convolution neural networks. In Proceedings of the 2014 13th International Conference on Machine Learning and Applications, ICMLA '14, page 159–164, USA. IEEE Computer Society.

Jaffe, D. L. (2003). Using augmented reality to improve walking in stroke survivors. In The 12th IEEE International Workshop on Robot and Human Interactive Communication (ROMAN), pages 79–83, Millbrae, CA, USA. IEEE.

Krizhevsky, A., Sutskever, I., and Hinton, G. E. (2017). Imagenet classification with deep convolutional neural networks. Commun. ACM, 60(6):84–90.

Leightley, D., Yap, M. H., Coulson, J., Barnouin, Y., and McPhee, J. S. (2015). Benchmarking human motion analysis using kinect one: An open source dataset. In 2015 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA), pages 1–7.

Lugaresi, C., Tang, J., Nash, H., McClanahan, C., Uboweja, E., Hays, M., Zhang, F., Chang, C.-L., Yong, M. G., Lee, J., Chang, W.-T., Hua, W., Georg, M., and Grundmann, M. (2019). MediaPipe: A framework for building perception pipelines. arXiv preprint arXiv:1906.08172.

Mehta, U. M. and Keshavan, M. S. (2015). Cognitive Rehabilitation and Modulating Neuroplasticity with Brain Stimulation: Promises and Challenges. Journal of Psychosocial Rehabilitation and Mental Health, 2(1):5–7.

Mukherjee, D., Levin, R. L., and Heller, W. (2006). The cognitive, emotional, and social sequelae of stroke: psychological and ethical concerns in post-stroke adaptation. Topics in stroke rehabilitation, 13(4):26–35.

Phan, H. L., Le, T. H., Lim, J. M., Hwang, C. H., and Koo, K.-i. (2022). Effectiveness of augmented reality in stroke rehabilitation: A meta-analysis. Applied Sciences, 12(4).

Rego, P., Moreira, P. M., and Reis, L. P. (2010). Serious games for rehabilitation: A survey and a classification towards a taxonomy. In 5th Iberian Conference on Information Systems and Technologies, pages 1–6, Santiago de Compostela, Spain. IEEE.

Rodrigues, L. G. S., Dias, D., Guimaraes, M. d. P., Brandao, A. F., Rocha, L., Iope, R. L., and Brega, J. R. F. (2021). Classification of human movements with motion capture data in a motor rehabilitation context. In Symposium on Virtual and Augmented Reality, SVR'21, page 56–63, New York, NY, USA. Association for Computing Machinery.

Simonyan, K. and Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556.

Toh, A., Jiang, L., and Lua, E. K. (2011). Augmented reality gaming for rehabhome. In Proceedings of the 5th International Conference on Rehabilitation Engineering & Assistive Technology, i-CREATe '11, Midview City, SGP. Singapore Therapeutic, Assistive & Rehabilitative Technologies (START) Centre.

World Stroke Organization (2019). Global stroke fact sheet. International Journal of Stroke.