Abstract
The modeling and online-generation of human-like body motion is a central topic in computer graphics and robotics. The analysis of the coordination structure of complex body movements in humans helps to develop flexible technical algorithms for movement synthesis. This chapter summarizes work that uses learned structured representations for the synthesis of complex human-like body movements in real-time. This work follows two different general approaches. The first one is to learn spatio-temporal movement primitives from human kinematic data, and to derive from this Dynamic Movement Primitives (DMPs), which are modeled by nonlinear dynamical systems. Such dynamical primitives are then coupled and embedded into networks that generate complex human-like behaviors online, as self-organized solutions of the underlying dynamics. The flexibility of this approach is demonstrated by synthesizing complex coordinated movements of single agents and crowds. We demonstrate that Contraction Theory provides an appropriate framework for the design of the stability properties of such complex composite systems. In addition, we demonstrate how such primitive-based movement representations can be embedded into a model-based predictive control architecture for the humanoid robot HRP-2. Using the primitive-based trajectory synthesis algorithm for fast online planning of full-body movements, we were able to realize flexibly adapting human-like multi-step sequences, which are coordinated with goal-directed reaching movements. The resulting architecture realizes fast online planing of multi-step sequences, at the same time ensuring dynamic balance during walking and the feasibility of the movements for the robot. The computation of such dynamically feasible multi-step sequences using state-of-the-art optimal control approaches would take hours, while our method works in real-time. The second presented framework for the online synthesis of complex body motion is based on the learning of hierarchical probabilistic generative models, where we exploit Bayesian machine learning approaches for nonlinear dimensionality reduction and the modeling of dynamical systems. Combining Gaussian Process Latent Variable Models (GPLVMs) and Gaussian Process Dynamical Models (GPDMs), we learned models for the interactive movements of two humans. In order to build an online reactive agent with controlled emotional style, we replaced the state variables of one actor by measurements obtained by real-time motion capture from a user and determined the most probable state of the interaction partner using Bayesian model inversion. The proposed method results in highly believable human-like reactive body motion.
J.P. Laumond et al. (Eds.): Geometric and Numerical Foundations of Movements, Springer STAR Series, 2016. © Springer-Verlag Berlin Heidelberg 2016.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
A. Ajoudani, J. Lee, A. Rocchi, M. Ferrati, E.M. Hoffman, A. Settimi, D.G. Caldwell, A. Bicchi, N.G. Tsagarakis, A manipulation framework for compliant humanoid COMAN: application to a valve turning task, in 14th IEEE-RAS International Conference on Humanoid Robots (Humanoids) (2014), pp. 664–670
O. Arikan, D.A. Forsyth, J.F. O’Brien, Motion synthesis from annotations. ACM Trans. Gr. SIGGRAPH ’03 22(3), 402–408 (2003)
C.G. Atkeson, A.W. Moore, S. Schaal, Locally weighted learning. A.I. Review 11, 11–73 (1997)
N.A. Bernstein, The Coordination and Regulation of Movements (Pergamon Press, New York, 1967)
C.M. Bishop, Pattern Recognition and Machine Learning (Springer, Berlin, 2007)
M. Brand, A. Hertzmann, Style machines, in Proceedings of SIGGRAPH Conference (2000), pp. 183–192
M. Brandao, L. Jamone, P. Kryczka, N. Endo, K. Hashimoto, A. Takanishi, Reaching for the unreachable: integration of locomotion and whole-body movements for extended visually guided reaching, in In Proceedings of 13th IEEE-RAS International Conference on Humanoid Robots (Humanoids) (2013), pp. 28–33
H. Carnahan, B.J. McFadyen, D.L. Cockell, A.H. Halverson, The combined control of locomotion and prehension. Neurosci. Res. Commun. 19, 91–100 (1996)
J. Chai, J.K. Hodgins, Performance animation from low-dimensional control signals. ACM Trans. Gr. SIGGRAPH ’05 24(3), 686–696 (2005)
C.-C. Chang, C.-J. Lin, LIBSVM: A Library for Support Vector Machines (2001). Software available at http://www.csie.ntu.edu.tw/~cjlin/libsvm
E. Chiovetto, A. d’Avella, D. Endres, M.A. Giese, A unifying algorithm for the identification of kinematic and electromyographic motor primitives, in Bernstein Conference (2013)
E. Chiovetto, M.A. Giese, Kinematics of the coordination of pointing during locomotion. PLoS One 8(11), e79555 (2013)
W. Daamen, S.P. Hoogendoorn, Controlled experiments to derive walking behaviour. Eur. J. Trans. Infrastruct. Res. 3(1), 39–59 (2003)
A. d’Avella, E. Bizzi, Shared and specific muscle synergies in neural motor behaviours. Proc. Natl. Acad. Sci. USA 102(8), 3076–3081 (2005)
S. Degallier, L. Righetti, S. Gay, A.J. Ijspeert, Towards simple control for complex, autonomous robotic applications: combining discrete and rhythmic motor primitives. Auton. Robots 31(2–3), 155–181 (2011)
A.W. Feng, Y. Xu, A. Shapiro, An example-based motion synthesis technique for locomotion and object manipulation. Proc. ACM SIGGRAPH I3D, 95–102 (2012)
T. Flash, B. Hochner, Motor primitives in vertebrates and invertebrates. Current Opinion Neurobiol. 15(6), 660–666 (2005)
A. Fod, M.J. Mataric, O.C. Jenkins, Automated derivation of primitives for movement classification. Auton. Robots 12(1), 39–54 (2002)
A. Gams, B. Nemec, L. Zlajpah, M. Wächter, A.J. Ijspeert, T. Asfour, A. Ude, Modulation of motor primitives using force feedback: Interaction with the environment and bimanual tasks, in In Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2013) (2013), pp. 5629–5635
M. Gienger, M. Toussaint, C. Goerick, Whole-body motion planning building blocks for intelligent systems, in Motion Planning for Humanoid Robots, ed. by K. Harada (Springer, Berlin, 2010), pp. 67–98
M.A. Giese, A. Mukovskiy, A. Park, L. Omlor, J.J.E. Slotine, Real-time synthesis of body movements based on learned primitives, in Statistical and Geometrical Approaches to Visual Motion Analysis. LNCS, vol. 5604, ed. by D. Cremers et al. (Springer, Berlin, 2009), pp. 107–127
M. Gleicher, Motion path editing, in Proceeding of 2001 ACM Symposium on Interactive 3D Graphics (2001), pp. 195–202
M. Gleicher, H.J. Shin, L. Kovar, A. Jepsen, Snap-together motion: assembling run-time animation. ACM Trans. Gr. SIGGRAPH ’03 22(3), 702–702 (2003)
K. Grochow, S.L. Martin, A. Hertzmann, Z. Popovic, Style-based inverse kinematics. ACM Trans. Gr. 23(3), 522–531 (2004)
D. Helbing, P. Molnár, I.J. Farkas, K. Bolay, Self-organizing pedestrian movement. Environ. Plan. B: Plan. Design 28, 361–383 (2001)
A. Herdt, H. Diedam, P.-B. Wieber, D. Dimitrov, K. Mombaur, M. Diehl, Online walking motion generation with automatic foot step placement. Adv. Robot. 24(5–6), 719–737 (2010)
E. Hsu, K. Pulli, J. Popovic, Style translation for human motion. ACM Trans. Gr. 24(3), 1082–1089 (2005)
Y. Huang, M. Kallmann, Planning motions for virtual demonstrators, in Intelligent Virtual Agents (Springer, Berlin, 2014), pp. 190–203
A.J. Ijspeert, J. Nakanishi, H. Hoffmann, P. Pastor, S. Schaal, Dynamical movement primitives: learning attractor models for motor behaviors. Neural Comput. 25(2), 328–373 (2013)
L. Ikemoto, O. Arikan, D.A. Forsyth, Generalizing motion edits with Gaussian processes. ACM Trans. Gr. 28(1), 1–12 (2009)
Y. Ivanenko, R. Poppele, F. Lacquaniti, Five basic muscle activation patterns account for muscle activity during human locomotion. J. Physiol. 556, 267–282 (2004)
S. Kajita, F. Kanehiro, K. Kaneko, K. Fujiwara, K. Harada, K. Yokoi, H. Hirukawa, Biped walking pattern generation by using preview control of zero-moment point, in Proceedings of International Conference on Robotics and Automation (2003), pp. 1620–1626
J. Koschorreck, K. Mombaur, Modeling and optimal control of human platform diving with somersaults and twists. Optim. Eng. 13(1), 29–56 (2012)
L. Kovar, M. Gleicher, F. Pighin, Motion graphs. Proc. SIGGRAPH 2002, 473–482 (2002)
S. Kuindersma, R. Deits, M. Fallon, A. Valenzuela, H. Dai, F. Permenter, T. Koolen, P. Marion, R. Tedrake, Optimization–based locomotion planning, estimation, and control design for the atlas humanoid robot. Auton. Robot. 1–27 (2015)
T. Kwon, K.H. Lee, J. Lee, S. Takahashi, Group motion editing. ACM Trans. Gr. SIGGRAPH 2008 27(3), 80–87 (2008)
W.M. Land, D.A. Rosenbaum, S. Seegelke, T. Schack, Whole-body posture planning in anticipation of a manual prehension task: prospective and retrospective effects. Acta Psychol. 114, 298–307 (2013)
M. Lau, Z. Bar-Joseph, J. Kuffner, Modeling spatial and temporal variation in motion data. ACM Trans. Gr. 28(5), Art.No.171 (2009)
N.D. Lawrence, Learning for larger datasets with the Gaussian process latent variable model. J. Mach. Learn. Res. - Proc. Track 2, 243–250 (2007)
N.D. Lawrence, R. Court, Local distance preservation in the GP-LVM through back constraints, in ICML (2006), pp. 513–520
A. Lerner, E. Fitusi, Y. Chrysanthou, D. Cohen-Or, Fitting behaviors to pedestrian simulations, in Proceedings of Eurographics/ACM SIGGRAPH Symposium on Computer Animation (2009), pp. 199–208
S. Levine, J.M. Wang, A. Haraux, Z. Popovi\(\acute{c}\), V. Koltun, Continuous character control with low-dimensional embeddings. ACM Trans. Gr. ACM SIGGRAPH 2012 31(4), Art.No.28 (2012)
Y. Li, T. Wang, H.Y. Shum, Motion texture: a two level statistical model for character motion synthesis. Proc. SIGGRAPH 2002, 465–472 (2002)
G. Liu, M. Xu, Z. Pan, A. El Rhalibi, Human motion generation with multifactor models. J. Comput. Anim. Virtual Worlds 22(4), 351–359 (2011)
W. Lohmiller, J.J.E. Slotine, On contraction analysis for nonlinear systems. Automatica 34(6), 683–696 (1998)
N. Mansard, O. Stasse, P. Evrard, A. Kheddar, A versatile generalized inverted kinematics implementation for collaborative working humanoid robots: the stack of tasks, in Proceedings of International Conference on Advanced Robotics (ICAR) (2009), p. art.119
R.G. Marteniuk, C.P. Bertram, Contributions of gait and trunk movement to prehension: perspectives from world- and body centered coordinates. Motor Control 5, 151–164 (2001)
M. Mühlig, M. Gienger, J.J. Steil, Human-robot interaction for learning and adaptation of object movements, in In Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2010) (2010), pp. 4901–4907
A. Mukovskiy, W. Land, T. Schack, M.A. Giese, Modeling of predictive human movement coordination patterns for applications in computer graphics. J. WSCG 23(2), 139–146 (2015)
A. Mukovskiy, A.-N. Park, L. Omlor, J.-J. Slotine, M.A. Giese, Self-organization of character behavior by mixing of learned movement primitives, in In Proceedings of the 13th Fall Workshop on Vision, Modeling, and Visualization (VMV) (2008), pp. 121–130
A. Mukovskiy, J.J.E. Slotine, M.A. Giese, Analysis and design of the dynamical stability of collective behavior in crowds. J. WSCG 19(1–3), 69–76 (2011)
A. Mukovskiy, J.J.E. Slotine, M.A. Giese, Dynamically stable control of articulated crowds. J. Comput. Sci. 4(4), 304–310 (2013)
A. Mukovskiy, C. Vassallo, M. Naveau, O. Stasse, P. Souères, M.A. Giese, Adaptive synthesis of dynamically feasible full-body movements for the humanoid robot HRP-2 by flexible combination of learned dynamic movement primitives. Robot. Auton. Syst. J. Comput. Sci. (submitted to) (2016)
R. Narain, A. Golas, S. Curtis, M. Lin, Aggregate dynamics for dense crowd simulation. ACM Trans. Gr. Art.122 28(5), 1–8 (2009)
M. Naveau, M. Kudruss, O. Stasse, C. Kirches, K. Mombaur, P. Souères, A reactive walking pattern generator based on nonlinear model predictive control. IEEE Robot. Autom. Lett. (2016) (in press)
L. Omlor, M.A. Giese, Anechoic blind source separation using Wigner marginals. J. Mach. Learn. Res. 12, 1111–1148 (2011)
D.A. Paley, N.E. Leonard, R. Sepulchre, D. Grunbaum, J.K. Parrish, Oscillator models and collective motion: spatial patterns in the dynamics of engineered and biological networks. IEEE Control Syst. Mag. 27, 89–105 (2007)
S. Paris, J. Pettré, S. Donikian, Pedestrian reactive navigation for crowd simulation: a predictive approach. Proc. Eurographics 2007 26(3), 665–674 (2007)
A. Park, A. Mukovskiy, L. Omlor, M.A. Giese, Self organized character animation based on learned synergies from full-body motion capture data, in Proceedings of International Conference on Cognitive Systems, (CogSys, 2008) (2008)
A. Park, A. Mukovskiy, L. Omlor, M.A. Giese, Synthesis of character behaviour by dynamic interaction of synergies learned from motion capture data, in The 16-th International Conference in Central Europe on Computer Graphics, Visualization and Computer Vision’2008, WSCG’08 (2008), pp. 9–16
A. Park, A. Mukovskiy, J.J.E. Slotine, M.A. Giese, Design of dynamical stability properties in character animation. Proc. VRIPHYS 09, 85–94 (2009)
S.I. Park, H.J. Shin, S.Y. Shin, On-line locomotion generation based on motion blending, in Proceedings of the 2002 ACM SIGGRAPH/Eurographics Symposium on Computer Animation (2002), pp. 105–111
N. Pelechano, J.M. Allbeck, N.I. Badler, Controlling individual agents in high-density crowd simulation, in Proceedings of Eurographics/ACM SIGGRAPH Symposium on Computer Animation (2007), pp. 99–108
Q.C. Pham, J.J.E. Slotine, Stable concurrent synchronization in dynamic system networks. Neural Netw. 20(3), 62–77 (2007)
C.L. Roether, L. Omlor, A. Christensen, M.A. Giese, Critical features for the perception of emotion from gait. J. Vis. 9(6), 15 (2009)
C. Rose, M. Cohen, B. Bodenheimer, Verbs and adverbs: multidimensional motion interpolation using radial basis functions. IEEE Comput. Gr. Appl. 18(5), 32–40 (1998)
C. Rose, B. Guenter, B. Bodenheimer, M. Cohen, Efficient generation of motion transitions using spacetime constraints, in Proceedings of ACM SIGGRAPH’96 International Conference on Computer Graphics and Interactive Techniques 30, 147–154 (1996)
D.A. Rosenbaum, Reaching while walking: reaching distance costs more than walking distance. Psychon. Bull. Rev. 15, 1100–1104 (2008)
D.A. Rosenbaum, R.G. Cohen, S.A. Jax, D.J. Weiss, R. van der Wel, The problem of serial order in behavior: Lashley’s legacy. Hum. Mov. Sci. 26(4), 525–554 (2007) (Europ, Workshop on Mov, Sci., 2007)
A. Safonova, J. Hodgins, N. Pollard, Synthesizing physically realistic human motion in low-dimensional, behavior-specific spaces. ACM Trans. Gr. 23(3), 514–521 (2004)
M. Santello, M. Flanders, J.F. Soechting, Postural hand synergies for tool use. J. Neurosci. 18(23), 10105–10115 (1998)
L. Scardovi, R. Sepulchre, Collective optimization over average quantities, in Proceedings of the 45th IEEE Conference on Decision and Control, San Diego, California (2006), pp. 3369–3374
S. Schaal, S. Kotosaka, D. Sternad, Nonlinear dynamical systems as movement primitives, in Proceedings of 1st IEEE-RAS International Conference on Humanoid Robots, Humanoids (Springer, Berlin, 2000), pp. 117–124
G. Schöner, M. Dose, C. Engels, Dynamics of behavior: theory and applications for autonomous robot architectures. Robot. Auton. Syst. 16(2–4), 213–245 (1995)
A. Shoulson, N. Marshak, M. Kapadia, N.I. Badler, ADAPT: the agent development and prototyping testbed. IEEE Trans. Vis. Comput. Gr. (TVCG) 99, 1–14 (2014)
M. Sreenivasa, P. Souères, J.-P. Laumond, Walking to grasp: modeling of human movements as invariants and an application to humanoid robotics. IEEE Trans. Syst. Man Cybern. Part A: Syst. Hum. 42(4), 880–893 (2012)
O. Stasse, Habilitation Thesis. Paul Sabatier University, CNRS, Toulouse (2013)
O. Stasse, B. Verelst, A. Davison, N. Mansard, F. Saidi, B. Vanderborght, C. Esteves, K. Yokoi, Integrating walking and vision to increase humanoid autonomy. Int. J. Humanoid Robot. Spec. Issue Cogn. Humanoid Robot. 5, 287–310 (2008)
M. Taïx, M.T. Tran, E. Souères, P. Guigon, Generating human-like reaching movements with a humanoid robot: a computational approach. J. Comput. Sci. 4, 269–284 (2013)
N. Taubert, A. Christensen, D. Endres, M.A. Giese, Online simulation of emotional interactive behaviors with hierarchical Gaussian Process Dynamical Models, in Proceedings of SAP’12 (ACM Press, New York, 2012), pp. 25–32
N. Taubert, D. Endres, A. Christensen, M.A. Giese, Shaking hands in latent space: modeling emotional interactions with Gaussian process latent variable models, in Proceedings of KI 2011: Advances in Artificial Intelligence, LNAI, ed. by S. Edelkamp, J. Bachpages (Springer, Berlin, 2011), pp. 330–334
N. Taubert, D. Endres, M.A. Giese, Reactive virtual reality avatar with controllable emotional style based on hierarchical Gaussian process dynamical models, in Proceedings of ICANN 2014 (2014), p. Art.No.25
N. Taubert, M. Löffler, N. Ludolph, A. Christensen, D. Endres, M.A. Giese, A virtual reality setup for controllable, stylized real-time interactions between humans and avatars with sparse Gaussian process dynamical models, in Proceedings of SAP’13 (2013), p. 41–44
D. Velychko, D. Endres, The variational Gaussian process dynamical model, in Proceedings of the Workshop on Advances in Approximate Bayesian Inference (NIPS, Montreal, Canada, 2015), pp. 1–6
D. Velychko, D. Endres, N. Taubert, M.A. Giese, Coupling Gaussian process dynamical models with product-of-experts kernels, in Proceedings of the 24th International Conference on Artificial Neural Networks. LNCS, vol. 8681 (Springer, Berlin, 2014), pp. 603–610
M. Vukobratovi\(\acute{c}\), Yu. Stepanenko, On the stability of anthropomorphic systems. Math. Biosci. 15, 1–37 (1972)
J.M. Wang, D.J. Fleet, A. Hertzmann, Multifactor Gaussian process models for style-content separation, in Proceedings of ICML (2007)
J.M. Wang, D.J. Fleet, A. Hertzmann, Gaussian process dynamical models for human motion. IEEE Trans. Pattern Anal. Mach. Intell. 30(2), 283–298 (2008)
W. Wang, J.J.E. Slotine, On partial contraction analysis for coupled nonlinear oscillators. Biol. Cybern. 92(1), 38–53 (2005)
Y. Wang, Z.-Q. Liu, L.-Z. Zhou, Learning style-directed dynamics of human motion for automatic motion synthesis, in Proceedings of IEEE International Conference on Systems, Man, and Cybernetics (2006), pp. 4428–4433
W.H. Warren, The dynamics of perception and action. Psychol. Rev. 113(2), 358–389 (2006)
M. Weigelt, T. Schack, The development of end-state comfort planning in preschool children. Exp. Psychol. 57(6), 476–782 (2010)
A.P. Witkin, Z. Popovi\(\acute{c}\), Motion warping. Proc. ACM SIGGRAPH’95 29, 105–108 (1995)
K. Yamane, Y. Nakamura, Dynamics filter - concept and implementation of on-line motion generator for human figures, in Proceedings of IEEE International Conference on Robotics and Automation (2000), pp. 688–695
Y. Ye, C.K. Liu, Synthesis of responsive motion using a dynamic model. Comput. Gr. Forum (Proc. Eurographics) 29(2), 555–562 (2010)
E. Yoshida, A. Mallet, F. Lamiraux, O. Kanoun, O. Stasse, M. Poirier, P-F. Dominey, J.-P. Laumond, K. Yokoi, ‘Give me the Purple Ball’ – he said to HRP-2 N.14, in Proceedings of IEEE-RAS International Conference on Humanoid Robots (Humanoids’07) (2007)
Acknowledgements
The work supported by EC FP7 under grant agreements FP7-611909 (Koroibot), H2020 ICT-644727 (CogIMon), FP7-604102 (HBP), PITN-GA-011-290011 (ABC), DFG GI 305/4-1, DFG GZ: KA 1258/15-1, DFG IRTG-GRK 1901 ‘The brain in action’, BMBF, FKZ: 01GQ1002A, and DFG SFB/TRR 135 Cardinal Mechanisms of Perception, project C06.
Author information
Authors and Affiliations
Corresponding authors
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2017 Springer International Publishing AG
About this chapter
Cite this chapter
Mukovskiy, A. et al. (2017). Modeling of Coordinated Human Body Motion by Learning of Structured Dynamic Representations. In: Laumond, JP., Mansard, N., Lasserre, JB. (eds) Geometric and Numerical Foundations of Movements . Springer Tracts in Advanced Robotics, vol 117. Springer, Cham. https://doi.org/10.1007/978-3-319-51547-2_11
Download citation
DOI: https://doi.org/10.1007/978-3-319-51547-2_11
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-51546-5
Online ISBN: 978-3-319-51547-2
eBook Packages: EngineeringEngineering (R0)