Publications & Patents

3rd Digital Heritage International Congress (DigitalHERITAGE) held jointly with 2018 24th International Conference on Virtual Systems & Multimedia (VSMM 2018)

Conventional approaches to a virtual heritage site that provide interpretations through curated content allow visitors only the lowest level of engagement, which is paying conscious, intentional attention to the content. In this paper, we propose a trajectory for visitor experience to a virtual heritage site that facilitates a higher level of engagement by letting visitors make their own interpretations. We developed a mobile virtual reality (VR) application that delivers this trajectory, which allows visitors to author mixed reality (MR) content that represents personal interpretations. The application provides three types of virtual assets to compose MR content: historical assets, emotional assets, and personal assets. We describe how visitors generally followed our trajectory and used virtual assets, engaging with virtual heritage and making interpretations. We related our findings to a discussion of how to support personal engagement and rich interpretation in a virtual heritage site.

3rd Digital Heritage International Congress (DigitalHERITAGE) held jointly with 2018 24th International Conference on Virtual Systems & Multimedia (VSMM 2018)

This paper proposes guidelines for the design of a Mixed Reality(MR) storytelling tool for cultural heritage sites that utilizes geotagged content and detailed location-specific narrative principles while prioritizing the intent of the author. Continuous efforts have been made to apply storytelling techniques in producing location-based digital heritage content over MR platforms. However, consideration for fine-tuned input parameters required at the authoring level has been consistently lacking. This has resulted in content that fails to identify, understand, and reflect the goals and needs of the author in maximizing the benefits of the narrative form for MR heritage experiences. To address this problem, we combine an analysis of existing location-based digital authoring tools with qualitative user studies conducted in a digital storytelling workshop. With the implications derived from our findings, we establish detailed design guidelines to provide a systematic narrative structure in the arrangement of geotagged content over various Points of Interest within the MR heritage space. Our study identifies two major authoring motivations for location-based MR storytelling: space-driven and story-driven. We thereby assert the need to bifurcate the design of the tool to support both these purposes and propose guidelines that differentiate the functions and task flow of each authoring mode.

3rd Digital Heritage International Congress (DigitalHERITAGE) held jointly with 2018 24th International Conference on Virtual Systems & Multimedia (VSMM 2018)

This study proposed a mobile virtual reality-based application for on-site risk management using Historical Building Information Modeling (HBIM) to alleviate the inefficiency of paper-based risk management of architectural heritage. In this research, we focused on the design of the metadata and the database structure that is based on the point of interest (PoI), anchor, and content metadata for advanced context-aware information retrieval from HBIM system in the mobile environment. To verify our method, we created a mobile VR-based application of the Gwandeokjeong building in Korea. After, we conducted a comparative experiment on performance time, task load and usability and proved the efficiency of our method for on-site risk management. In conclusion, our metadata and database structure in this study contribute to suggest a method to interoperate between HBIM system and mobile VR system and conduct advanced context-aware information retrieval.

3rd Digital Heritage International Congress (DigitalHERITAGE) held jointly with 2018 24th International Conference on Virtual Systems & Multimedia (VSMM 2018)

Despite various efforts to provide visually immersive experiences and various information about virtual cultural heritage sites, few studies have investigated how to effectively provide media related to cultural heritage sites in virtual reality or to measure the effectiveness of the virtual tourism experience in improving the affect and visit intention to the actual tourist destination. In this study, we examined the effects of a video-based virtual tour, applied via film-induced tourism, on post-VR attitude and behavioral visit intention to a cultural heritage site compared with those of a basic virtual tour. We also aimed to determine how to effectively provide broadcast content as a storytelling medium in a virtual reality tour, considering characteristics of a cultural heritage site. Through the experiment, we proved that video-based virtual experience is more effective in improving visitors' positive attitude and visit intention than the basic virtual tour, which was nevertheless effective as well. We also derived design implications for the utilization of location-based broadcast video clips.

IEEE International Symposium on Mixed and Augmented Reality (ISMAR) 2018Top-tier

We propose a novel method for model-based 3D tracking of hand articulations that is effective even for fastmoving hand postures in depth images. A large number of augmented reality (AR) and virtual reality (VR) studies have used model-based approaches for estimating hand postures and tracking movements. However, these approaches exhibit limitations if the hand moves rapidly or into the camera's field of view. To overcome these problems, researchers attempted a hybrid strategy that uses multiple initializations for 3D tracking of articulations. However, this strategy also exhibits limitations. For example, in genetic optimization, the hypotheses generated from the previous solution may search for a solution in an incorrect search space in a fast-moving hand gesture. This problem also occurs if the search space selected from the results of a trained model does not cover the true solution although the tracked hand moves slowly. Our proposed method estimates the hand pose based on model-based tracking guided by classification and search space adaptation. From the classification by a convolutional neural network (CNN), a data-driven prior is included in the objective function and additional hypotheses are generated in particle swarm optimization (PSO). In addition, the search spaces of the two sets of the hypotheses, generated by the data-driven prior and the previous solution, are adaptively updated using the distribution of each set of the hypotheses. We demonstrated the effectiveness of the proposed method by applying it to an American Sign Language (ASL) dataset consisting of fast-moving hand postures. The experimental results demonstrate that the proposed algorithm exhibits more accurate tracking results compared to other state-of-the-art tracking algorithms.

IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct) 2018

In this demonstration, we will show a prototype system with sensor fusion approach to robustly track 6 degrees of freedom of hand movement and support intuitive hand gesture interaction and 3D object manipulation for Mixed Reality head-mounted displays. Robust tracking of hand and finger with egocentric camera remains a challenging problem, especially with self-occlusion - for example, when user tries to grab a virtual object in midair by closing the palm. Our approach leverages the use of a common smart watch worn on the wrist to provide a more reliable palm and wrist orientation data, while fusing the data with camera to achieve robust hand motion and orientation for interaction.

IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct) 2018

We propose a method that is effective in tracking 3D hand poses occluded by a real object. Since existing model-based tracking methods rely only on observed images to estimate hand joints, tracking generally fails when the hand joints are largely invisible. This problem becomes more prevalent when the tracked hand is grabbing an object, as occlusion by the object makes it harder to find a proper correspondence between the hand model and observation. The proposed method utilizes the occluded part of the hand as additional information for model-based tracking. The occluded depth information is reconstructed according to the geometric of the object and model-based tracking is employed based on particle swarm optimization (PSO). We demonstrate that the reconstructed depth information improves the performance of tracking an object-grabbing hand.

Adjunct Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology

Augmented collaboration in a shared house design scenario has been studied widely with various approaches. However, those studies did not consider human perception. Our goal is to lower the user's perceptual load for augmented collaboration in shared space design scenarios. Applying attention theories, we implemented shared head gaze, shared selected object, and collaborative manipulation features in our system in two different versions with HoloLens. To investigate whether user perceptions of the two different versions differ, we conducted an experiment with 18 participants (9 pairs) and conducted a survey and semi-structured interviews. The results did not show significant differences between the two versions, but produced interesting insights. Based on the findings, we provide design guidelines for collaborative AR systems.

In International Conference on Human Interface and the Management of Information, Springer, Cham, 2018

In applying MR (AR/VR) technologies to cultural heritage sites, the design and implementation of mobile MR applications have mostly lacked a holistic and systematic approach in the viewpoint of retrieving existing content and providing personalized experiences to visitors on the fly. To address this issue, we designed and implemented the KCTM application in which various types of content from different databases are brought together and reorganized by a newly proposed metadata schema that assigns spatio-temporal information to them for the purpose of visualizing them in MR environments. According to the user evaluation, most of the participants expressed strong interest in touring the heritage sites augmented with various related content.

International Conference on Human Interface and the Management of Information (HIMI), pp. 188-201

As the popularization of augmented human, the influence of computing devices on human life is growing. We are facing lots of service in daily life, and it will be more in future. As a result, the border between real and virtual life is vanishing. To manage and maintain yourself in the flood of information, we need to know yourself in detail and quickly. For this reason, there were several research studies about the Quantified Self. However, the studies and applications of QS were held in separately by focusing each limited information, and it arises limitation on understanding in bird-eye view. To address these issues, we propose the unified concept of the Holistic Quantified Self (HQS) with its framework. HQS consists of the quantified indexes of user status which includes physical, emotional, social, and digital behavioral state. By collecting and quantifying the user’s multi-modal sensor data, HQS can be generated. Then, interpreting the pattern of HQS could be utilized to provide customized services to users. HQS framework describe an overall process to ensure the personalized service. Through these study, we expect to fulfill augmented human which enhances user abilities and contribute quality of social life through the development of the personal companion.