Abstract
Modeling multimodal face-to-face interaction is a crucial step in the process of building social robots or users-aware Embodied Conversational Agents (ECA). In this context, we present a novel approach for human behavior analysis and generation based on what we called “Incremental Discrete Hidden Markov Model” (IDHMM). Joint multimodal activities of interlocutors are first modeled by a set of DHMMs that are specific to supposed joint cognitive states of the interlocutors. Respecting a task-specific syntax, the IDHMM is then built from these DHMMs and split into i) a recognition model that will determine the most likely sequence of cognitive states given the multimodal activity of the interlocutor, and ii) a generative model that will compute the most likely activity of the speaker given this estimated sequence of cognitive states. Short-Term Viterbi (STV) decoding is used to incrementally recognize and generate behavior. The proposed model is applied to parallel speech and gaze data of interacting dyads.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
Similar content being viewed by others
References
Otsuka, K.: Multimodal Conversation Scene Analysis for Understanding People’s Communicative Behaviors in Face-to-Face Meetings, pp. 171–179 (2011)
Scherer, S., Marsella, S., Stratou, G., Xu, Y., Morbini, F., Egan, A., Morency, L.-P.: Perception markup language: towards a standardized representation of perceived nonverbal behaviors. In: Intelligent Virtual Agents, pp. 455–463 (2012)
Argyle, M.: Bodily Communication. Taylor & Francis (1975)
Lakin, J.L., Jefferis, V.E., Cheng, C.M., Chartrand, T.L.: The Chameleon Effect as Social Glue: Evidence for the Evolutionary Significance of Nonconscious Mimicry. Journal of Nonverbal Behavior 27(3), 145–162 (2003)
Kopp, S.: Social resonance and embodied coordination in face-to-face conversation with artificial interlocutors. Speech Commun. 52(6), 587–597 (2010)
Bailly, G., Raidt, S., Elisei, F.: Gaze, conversational agents and face-to-face communication. Speech Communication 52(6), 598–612 (2010)
Gatica-Perez, D.: Automatic nonverbal analysis of social interaction in small groups: A review. Image and Vision Computing 27(12), 1775–1787 (2009)
Otsuka, K., Sawada, H., Yamato, J.: Automatic inference of cross-modal nonverbal interactions in multiparty conversations: ‘who responds to whom, when, and how?’ from gaze, head gestures, and utterances. In: Proceedings of the 9th International Conference on Multimodal Interfaces, New York, NY, USA, pp. 255–262 (2007)
Zhang, D., Gatica-Perez, D., Bengio, S., McCowan, I.: Modeling individual and group actions in meetings with layered HMMs. IEEE Transactions on Multimedia 8(3), 509–520 (2006)
Salamin, H., Vinciarelli, A.: Automatic Role Recognition in Multiparty Conversations: An Approach Based on Turn Organization, Prosody, and Conditional Random Fields. IEEE Transactions on Multimedia 14(2), 338–345 (2012)
Gatica-Perez, D.: Analyzing group interactions in conversations: a review. In: 2006 IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems, pp. 41–46 (2006)
Cassell, J., Vilhjalmsson, H., Bickmore, T.: BEAT: The Behavior Expression Animation Toolkit (2001)
Krenn, B.: The NECA project: Net environments for embodied emotional conversational agents. In: Proc. of Workshop on Emotionally Rich Virtual Worlds with Emotion Synthesis at the 8th International Conference on 3D Web Technology (Web3D), vol. 35. St. Malo, France (2003)
Krenn, B., Pirker, H.: Defining the gesticon: Language and gesture coordination for interacting embodied agents. In: Proc. of the AISB-2004 Symposium on Language, Speech and Gesture for Expressive Characters, pp. 107–115 (2004)
Kopp, S., Jung, B., Lessmann, N., Wachsmuth, I.: Max - A Multimodal Assistant in Virtual Reality Construction. KI 17(4), 11 (2003)
Kopp, S., Krenn, B., Marsella, S.C., Marshall, A.N., Pelachaud, C., Pirker, H., Thórisson, K.R., Vilhjálmsson, H.H.: Towards a Common Framework for Multimodal Generation: The Behavior Markup Language. In: Gratch, J., Young, M., Aylett, R.S., Ballin, D., Olivier, P. (eds.) IVA 2006. LNCS (LNAI), vol. 4133, pp. 205–217. Springer, Heidelberg (2006)
Vilhjálmsson, H., Cantelmo, N., Cassell, J., Chafai, N.E., Kipp, M., Kopp, S., Mancini, M., Marsella, S., Marshall, A., Pelachaud, C.: The behavior markup language: Recent developments and challenges. In: Intelligent Virtual Agents, pp. 99–111 (2007)
Heylen, D., Kopp, S., Marsella, S.C., Pelachaud, C., Vilhjálmsson, H.H.: The Next Step towards a Function Markup Language. In: Prendinger, H., Lester, J.C., Ishizuka, M. (eds.) IVA 2008. LNCS (LNAI), vol. 5208, pp. 270–280. Springer, Heidelberg (2008)
Thiebaux, M., Marsella, S., Marshall, A.N., Kallmann, M.: Smartbody: Behavior realization for embodied conversational agents. In: Proceedings of the 7th International Joint Conference on Autonomous Agents and Multiagent Systems, vol. 1, pp. 151–158 (2008)
Le, Q.A., Pelachaud, C.: Generating Co-speech Gestures for the Humanoid Robot NAO through BML. In: Efthimiou, E., Kouroupetroglou, G., Fotinea, S.-E. (eds.) GW 2011. LNCS, vol. 7206, pp. 228–237. Springer, Heidelberg (2012)
Bailly, G.: Boucles de perception-action et interaction face-à-face. Revue Fran\ccaise De Linguistique Appliquée 13(2), 121–131 (2009)
Rabiner, L.R.: A tutorial on hidden markov models and selected applications in speech recognition. In: Proceedings of the IEEE, pp. 257–286 (1989)
Seward, A.: Low-Latency Incremental Speech Transcription in the Synface Project
Ryynänen, M., Klapuri, A.: Automatic Bass Line Transcription from Streaming Polyphonic Audio. In: Proceedings of the 2007 IEEE International Conference on Acoustics, Speech, and Signal Processing, pp. 1437–1440 (2007)
Lou, Y., Zhang, C., Zheng, Y., Xie, X., Wang, W., Huang, Y.: Map-matching for low-sampling-rate GPS trajectories. In: Proceedings of the 17th ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems, New York, NY, USA, pp. 352–361 (2009)
Yuan, J., Zheng, Y., Zhang, C., Xie, X., Sun, G.-Z.: An Interactive-Voting Based Map Matching Algorithm, pp. 43–52 (2010)
Šrámek, R., Brejová, B., Vinař, T.: On-line Viterbi Algorithm and Its Relationship to Random Walks. arXiv:0704.0062 (March 2007)
Bloit, J., Rodet, X.: Short-time Viterbi for online HMM decoding: Evaluation on a real-time phone recognition task. In: IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2008, pp. 2121–2124 (2008)
Goh, C.Y., Dauwels, J., Mitrovic, N., Asif, M.T., Oran, A., Jaillet, P.: Online map-matching based on Hidden Markov model for real-time traffic sensing applications. In: 2012 15th International IEEE Conference on Intelligent Transportation Systems (ITSC), pp. 776–781 (2012)
HTK, The Hidden Markov Model Toolkit, http://htk.eng.cam.ac.uk/
Dunham, M., Murphy, K.: PMTK3: Probabilistic modeling toolkit for Matlab/Octave, http://code.google.com/p/pmtk3/
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2013 Springer International Publishing Switzerland
About this paper
Cite this paper
Mihoub, A., Bailly, G., Wolf, C. (2013). Social Behavior Modeling Based on Incremental Discrete Hidden Markov Models. In: Salah, A.A., Hung, H., Aran, O., Gunes, H. (eds) Human Behavior Understanding. HBU 2013. Lecture Notes in Computer Science, vol 8212. Springer, Cham. https://doi.org/10.1007/978-3-319-02714-2_15
Download citation
DOI: https://doi.org/10.1007/978-3-319-02714-2_15
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-02713-5
Online ISBN: 978-3-319-02714-2
eBook Packages: Computer ScienceComputer Science (R0)