Real-time computer vision system for tracking simultaneously subject-specific rigid head and non-rigid facial mimic movements using a contactless sensor and system of systems approach

Tan-Nhu Nguyen (1, 2) , Stéphanie Dakpé (3, 4) , Marie-Christine Ho Ba Tho (2) , Tien-Tuan Dao (5, 2)

1 Ho Chi Minh city University of Technology and Education
2 BMBI - Biomécanique et Bioingénierie
3 CHU Amiens-Picardie
4 CHIMERE - CHirurgie, IMagerie et REgénération tissulaire de l’extrémité céphalique - Caractérisation morphologique et fonctionnelle - UR UPJV 7516
5 LaMcube - Laboratoire de Mécanique, Multiphysique, Multiéchelle - UMR 9013

Background and Objective: Head and facial mimic animations play important roles in various fields such as human-machine interactions, internet communications, multimedia applications, and facial mimic analysis. Numerous studies have been trying to simulate these animations. However, they hardly achieved all requirements of full rigid head and non-rigid facial mimic animations in a subject-specific manner with real-time framerates. Consequently, this present study aimed to develop a real-time computer vision system for tracking simultaneously rigid head and non-rigid facial mimic movements. Methods: Our system was developed using the system of systems approach. A data acquisition sub-system was implemented using a contactless Kinect sensor. A subject-specific model generation sub-system was designed to create the geometrical model from the Kinect sensor without texture information. A subject-specific texture generation sub-system was designed for enhancing the reality of the generated model with texture information. A head animation sub-system with graphical user interfaces was also developed. Model accuracy and system performances were analyzed. Results: The comparison with MRI-based model shows a very good accuracy level (distance deviation of similar to 1 mm in neutral position and an error range of [2-3 mm] for different facial mimic positions) for the generated model from our system. Moreover, the system speed can be optimized to reach a high framerate (up to 60 fps) during different head and facial mimic animations. Conclusions: This study presents a novel computer vision system for tracking simultaneously subject-specific rigid head and non-rigid facial mimic movements in real time. In perspectives, serious game technology will be integrated into this system towards a full computer-aided decision support system for facial rehabilitation. (C) 2020 Elsevier B.V. All rights reserved.

Domaines: Sciences du Vivant [q-bio]

Tan-Nhu Nguyen, Stéphanie Dakpé, Marie-Christine Ho Ba Tho, Tien-Tuan Dao. Real-time computer vision system for tracking simultaneously subject-specific rigid head and non-rigid facial mimic movements using a contactless sensor and system of systems approach. Computer Methods and Programs in Biomedicine, 2020, 191, ⟨10.1016/j.cmpb.2020.105410⟩. ⟨hal-03599052⟩