3D motion capture by computer vision and virtual rendering

David Antonio Gómez Jáuregui



Networked 3D virtual environments allow multiple users to interact with each other over the Internet. Users can share some sense of telepresence by remotely animating an avatar that represents them. However, avatar control may be tedious and still render user gestures poorly. This work aims at animating a user’s avatar from real time 3D motion capture by monoscopic computer vision, thus allowing virtual telepresence to anyone using a personal computer with a webcam.

The approach followed consists of registering a 3D articulated upper-body model to a video sequence. This involves searching iteratively for the best match between features extracted from the 3D model and from the image. A two-step registration process matches regions and then edges. The first contribution of this thesis is a method of allocating computing iterations under real-time constrain that achieves optimal robustness and accuracy.

The major issue for robust 3D tracking from monocular images is the 3D/2D ambiguities that result from the lack of depth information. Particle filtering has become a popular framework for propagating multiple hypotheses between frames. As a second contribution, this thesis enhances particle filtering for 3D/2D registration under limited computation constrains with a number of heuristics, the contribution of which is demonstrated experimentally. A parameterization of the arm pose based on their end-effector is proposed to better model uncertainty in the depth direction. Finally, evaluation is accelerated by computation on GPU.

In conclusion, the proposed algorithm is demonstrated to provide robust real-time 3D body tracking from a single webcam for a large variety of gestures including partial occlusions and motion in the depth direction.


 Ph.D Thesis

·       David Antonio Gómez Jáuregui, “3D Motion Capture by Computer Vision and Virtual Rendering”, PhD Thesis 2011TELE0015 prepared at Télécom SudParis defended on 4 May 2011.


Thesis defense (power point presentation)

·       Acquisition 3D des gestes par vision artificielle et restitution virtuelle



·        David Antonio Gomez Jauregui, Patrick Horain, Manoj Kumar Rajagopal, Senanayak Sesh Kumar Karri. Real-Time Particle Filtering with Heuristics for 3D Motion Capture by Monocular Vision, IEEE International Workshop on Multimedia Signal Processing 2010 (MMSP'10), Saint-Malo, France, October 4-6, 2010, pp. 139 – 144.


·        David Antonio Gomez Jauregui, Patrick Horain, « Acquisition 3D des gestes par vision artificielle et restitution virtuelle ». A3DM '10 : Journée scientifique du colloque "Analyse 3d du mouvement", 17-18 juin 2010, Poitiers, France, 2010


·        Patrick Horain, José Marques Soares, Dianle Zhou, Zhenbo Li, David Antonio Gomez Jauregui, Yannick Allusse, “Perceiving and rendering users in a 3D interaction”, Proceedings of the Second IEEE International Conference on Intelligent Human Computer Interaction (IHCI 2010), January 16-18, 2010, Allahabad, India, Springer (ISBN 978-81-8489-540-7), pp. 42-53.


·        Zhenbo Li, Jun Yue, David Antonio Gómez Jáuregui, "A new virtual reality environment used for e-Learning", IEEE International Symposium on IT in Medicine & Education, 14-16 August 2009 (ITIME '09 external), Vol. 1, p. 445-449.


·        David Antonio Gómez Jáuregui, Patrick Horain, « Recalage sur les contours et recalage sur les régions pour l'acquisition 3D des gestes en temps réel par vision monoscopique », Actes en ligne d'ORASIS'09 - Congrčs des jeunes chercheurs en vision par ordinateur, Trégastel, France, 8 au 12 juin 2009.


·        David Antonio Gómez Jáuregui, Patrick Horain, "Region-based vs. edge-based registration for 3D motion capture by real time monoscopic vision", Proceedings of MIRAGE 2009, 4-6 May, 2009, INRIA Rocquencourt, France, A. Gagalowicz and W. Philips (Eds.), LNCS 5496, Springer-Verlag, 2009, pp. 344–355.


·        David Antonio Gómez Jáuregui, Patrick Horain & Fawaz Baroud, « Acquisition 3D des gestes par vision monoscopique en temps réel », Actes de MajecSTIC 2008, Marseille, 29 au 31 octobre 2008.


Videos and demos

                                  Demo 1                                                                           Demo 2                          

                          demo_david_gestes1.jpg                          marques.jpg


Demo 3                                                                         Demo 4

                         demo_david_gestes2.jpg                         interview.jpg