IPS: Integrating Pose with Speech for enhancement of body pose estimation in VR remote collaboration

Abstract

We propose a Speech-Pose integration method to overcome the limitation of existing body pose estimation. Unlike previous Speech-based gesture generation method, our proposal reflects the user's actual pose using a vision-based system and speech as a subsidiary. When the system detects the camera out of sight, we carry out a context-aware method analyzing the speech. The system determines the target pose and connects it with the last pose captured by the camera using the bounding box. Our system can be a robust solution for avatar-mediated remote collaboration which requires accurate gesture delivery such as VR remote yoga training.