Abstract
In recent years, online learning plays an essential part in education due to distance learning technology development and control of COVID-19. In this context, engagement, a mental state to enhance the learning process, has been brought into the limelight. However, the existing engagement datasets are of a small scale and not suitable for education time-series research. We proposed an estimation method on time-series face and body features captured by built-in PC cameras to improve the engagement estimation on small and irregularly wild datasets. We designed upper body features using the facial and body key points extracted from OpenPose. To reduce the influence of the extracted noises from OpenPose, the moving average, the average value of a fixed period in the videos, is used to process the training data. Then, we compose a time-series dataset of online tasks with 19 participants. In the composed dataset, there remained self-reports of participants’ mental state and external observation to confirm the different engagement levels in the answering process. The combined self-reports and external observation results were used as the engagement label. Finally, the transfer learning was used to solve the insufficient data issue. We pre-trained a long short-term memory (LSTM) sequence deep learning model on a big dataset and transferred the trained model to share learned feature extraction and retrain our dataset. Our proposed method achieved 63.7% in experiments and could apply to estimate and detection engagement in future works.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Wong, J., Baars, M., Davis, D., Van Der Zee, T., Houben, G.J., Paas, F.: Supporting self-regulated learning in online learning environments and MOOCs: a systematic review. In: Int. J. Hum.-Comput. Interact. 35(4-5), 356–373 (2019)
Dhawan, S.: Online learning: a panacea in the time of COVID-19 Crisis. J. Educ. Technol. Syst. 49(1), 5–22 (2020). https://doi.org/10.1177/0047239520934018
Tran, M.T., Hasegawa, S.: Self-regulated learning recognition and improvement framework. In: The Asian Conference on Education 2020: Official Conference Proceedings, pp. 449–465. (2021)
Hasegawa, S., Hirako, A., Zheng, X.W., Karimah, S.N., Ota, K., Unoki, T.: Learner’s mental state estimation with PC built-in camera. In: International Conference on Human-Computer Interaction, HCII 2020: Learning and Collaboration Technologies. Human and Technology Ecosystems, vol. 12206, pp. 165–175 (2020)
Chang, C., Zhang, C., Chen, L., Liu, Y.: An ensemble model using face and body tracking for engagement detection. In: The 20th ACM International Conference on Multimodal Interaction, pp. 616–622 (2018)
Gupta, A.., D’Cunha, A., Awasthi, K., Balasubramanian, V.: DAiSEE: towards User Engagement Recognition in the Wild. J. Latex Class Files 14(8), (2015) https://doi.org/10.10007/1234567890
Dewan, M. A. A., Lin, F., Wen, D., Uddin, Z.: A deep learning approach to detecting engagement of online learners. In: 2018 IEEE Smart World, Ubiquitous Intelligence Computing, Advanced Trusted Computing, Scalable Computing Communications, Cloud Big Data Computing, Internet of People and Smart City Innovation, pp. 1895–1902 (2018)
Fawaz, H.I., Forestier, G., Weber, J., Idoumghar, L., Muller, P.: Transfer learning for time series classification. arXiv:1811.01533v1 [cs.LG] 5 Nov 2018
Nomura, K., Augereau, O.: Poster: estimation of student’s engagement based on the posture. In: Proceedings of the 2019 ACM International Symposium on Wearable Computers, pp. 164–167 (2019)
Monkaresi, H., Bosch, N., Calvo, R.A.: Automated detection of engagement using video-based estimation of facial expressions and heart rate. IEEE Trans. Affect. Comput. 8(1), 15–28 (2017)
Kleinsmith, A., Berthouze, N.B.: Affective body expression perception and recognition: a survey. IEEE Trans. Affect. Comput. 14–18 (2019)
Dhall, A.: EmotiW 2019: automatic emotion, engagement and cohesion prediction tasks. In: 2019 International Conference on Multimodal Interaction (ICMI 2019), pp. 546–550 (2016)
Nordlund, A., Pahlsson, L., Holmberg, C., Lind, K., Wallin, A.: The cognitive assessment battery (CAB): a rapid test of cognitive domains. Int. Psychogeriatr. 23(7), 1144–1151 (2011)
Fredricks, J.A., Blumenfeld, P.C., Paris, A.H.: School engagement: potential of the concept, state of the evidence. Rev. Educ. Res. 74(1), 59–109 (2004)
Cao, Z., Hidalgo, G., Simon, T., Wei, S., Sheikh, Y.: OpenPose: Realtime Multi-person 2D pose estimation using part affinity fields. IEEE Trans. Pattern Anal. Mach. Intell. 43(1), 172–186 (2019)
Dewan, M.A.A., Lin, F.: Engagement detection in online learning: a review. Smart Learn. Environ. 6(1), 1–20 (2019)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Zheng, X., Hasegawa, S., Tran, MT., Ota, K., Unoki, T. (2021). Estimation of Learners’ Engagement Using Face and Body Features by Transfer Learning. In: Degen, H., Ntoa, S. (eds) Artificial Intelligence in HCI. HCII 2021. Lecture Notes in Computer Science(), vol 12797. Springer, Cham. https://doi.org/10.1007/978-3-030-77772-2_36
Download citation
DOI: https://doi.org/10.1007/978-3-030-77772-2_36
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-77771-5
Online ISBN: 978-3-030-77772-2
eBook Packages: Computer ScienceComputer Science (R0)