Gabyong Park

Previous methods successfully estimated 3D hand poses from unblurred depth images with slow and smooth hand motions. However, the performance drops when the depth images are contaminated by motion blur due to fast hand motion. In this paper, we exploit an infrared (IR) image input, which is weakly blurred under fast hand motion. The proposed method is based on domain transfer learning from depth to infrared images. Note we do not have IR images with hand skeletons, thus proposing self-supervision rather than direct supervision using the skeleton labels. We train a Hand Image Generator (HIG) and two Hand Pose Estimators (HPEs) on paired depth and infrared images via self-supervision using a consistency loss, guided by an existing HPE trained on paired depth and hand skeleton entries. The IR-based HPE is then refined on the weakly blurred infrared images. The qualitative and quantitative experiments demonstrate that the proposed method accurately estimates 3D hand poses under motion blur by fast hand motion, while existing depth-based methods fail. Our solution therefore supports fast 3D manipulation of virtual objects for augmented reality applications. Our model and dataset are publicly available for future research.1

We present a sensor-fusion method that exploits a depth camera and a gyroscope to track the articulation of a hand in the presence of excessive motion blur. In case of slow and smooth hand motions, the existing methods estimate the hand pose fairly accurately and robustly, despite challenges due to the high dimensionality of the problem, self-occlusions, uniform appearance of hand parts, etc. However, the accuracy of hand pose estimation drops considerably for fast-moving hands because the depth image is severely distorted due to motion blur. Moreover, when hands move fast, the actual hand pose is far from the one estimated in the previous frame, therefore the assumption of temporal continuity on which tracking methods rely, is not valid. In this paper, we track fast-moving hands with the combination of a gyroscope and a depth camera. As a first step, we calibrate a depth camera and a gyroscope attached to a hand so as to identify their time and pose offsets. Following that, we fuse the rotation information of the calibrated gyroscope with model-based hierarchical particle filter tracking. A series of quantitative and qualitative experiments demonstrate that the proposed method performs more accurately and robustly in the presence of motion blur, when compared to state of the art algorithms, especially in the case of very fast hand rotations.

We propose a novel method for model-based 3D tracking of hand articulations that is effective even for fastmoving hand postures in depth images. A large number of augmented reality (AR) and virtual reality (VR) studies have used model-based approaches for estimating hand postures and tracking movements. However, these approaches exhibit limitations if the hand moves rapidly or into the camera's field of view. To overcome these problems, researchers attempted a hybrid strategy that uses multiple initializations for 3D tracking of articulations. However, this strategy also exhibits limitations. For example, in genetic optimization, the hypotheses generated from the previous solution may search for a solution in an incorrect search space in a fast-moving hand gesture. This problem also occurs if the search space selected from the results of a trained model does not cover the true solution although the tracked hand moves slowly. Our proposed method estimates the hand pose based on model-based tracking guided by classification and search space adaptation. From the classification by a convolutional neural network (CNN), a data-driven prior is included in the objective function and additional hypotheses are generated in particle swarm optimization (PSO). In addition, the search spaces of the two sets of the hypotheses, generated by the data-driven prior and the previous solution, are adaptively updated using the distribution of each set of the hypotheses. We demonstrated the effectiveness of the proposed method by applying it to an American Sign Language (ASL) dataset consisting of fast-moving hand postures. The experimental results demonstrate that the proposed algorithm exhibits more accurate tracking results compared to other state-of-the-art tracking algorithms.

In this paper, we present a unified framework for remote collaboration using interactive augmented reality (AR) authoring and hand tracking methods. The proposed framework enables a local user to organize AR digital contents for making a shared working environment and collaborate multiple users in the distance. To develop the framework, we combine two core technologies: (i) interactive AR authoring method utilizing a smart input device for making a shared working space, (ii) hand-augmented object interaction method by tracking two hands in egocentric camera view. We implement a prototype of the proposed remote collaboration framework for testing its feasibility in an indoor environment. To the end, we expect that our framework enables collaboration as feeling a sense of co-presence with remote users in a user’s friendly AR working space.

We propose an efficient method for model-based 3D tracking of hand articulations observed from an egocentric viewpoint that aims at supporting the manipulation of virtual objects. Previous model-based approaches optimize non-convex objective functions defined in the 26 Degrees of Freedom (DoFs) space of possible hand articulations. In our work, we decompose this space into six articulation subspaces (6 DoFs for the palm and 4 DoFs for each finger). We also label each finger with a Gaussian model that is propagated between successive image frames. As confirmed by a number of experiments, this divide-and-conquer approach tracks hand articulations more accurately than existing model-based approaches. At the same time, real time performance is achieved without the need of GPGPU processing. Additional experiments show that the proposed approach is preferable for supporting the accurate manipulation of virtual objects in VR/AR scenarios.

The present invention relates to a hand position estimation method and system for estimating a 3-dimensional hand position for quick hand movement, on the basis of domain transfer learning from a depth image to an infrared image, the hand position estimation method comprising the steps of: processing a depth image and an infrared image for hand movement; synthesizing a depth map with the infrared image by using a hand image generator (HIG), and estimating skeletal positions of hand joints from each of the depth map and an infrared map; and calculating a 3-dimensional hand position by using the skeletal positions and the center of a hand depth image.