Publications & Patents

Proceedings of the 2026 CHI Conference on Human Factors in Computing SystemsTop-tier

Question-asking is one of the key indicators of cognitive engagement. However, understanding how the distinct psychological affordances of presentation media shape learners’ spoken inquiries with embodied Intelligent Virtual Agents (IVAs) remains limited. To systematically examine this process, we propose a 5W1H-based framework for analyzing learner questions. Using this framework, we conducted a user study comparing an Augmented Reality-based IVA (AR-IVA) deployed in the physical environment with a screen-based IVA (Video-IVA) during cardiopulmonary resuscitation (CPR) instruction. Results showed that the AR-IVA elicited higher spatial and social presence and promoted more frequent and longer questions focused on clarification and understanding. In contrast, the Video-IVA encouraged questions regarding procedural refinement. Presence acted as a selective filter, shaping the timing and …

IEEE Conference on Virtual Reality and 3D User Interfaces (VR) 2026Top-tier

We propose a novel object-mediated hand interaction system that enables real-time operation with everyday objects on wearable augmented reality (AR) devices. Despite recent advances, both commercial and academic hand interaction techniques remain constrained, typically requiring external hardware or depending exclusively on bare-hand gestures. Motivated by these constraints, we developed an offloading framework that integrates a high-fidelity transformer-based 3D hand reconstruction model with a dynamic gesture recognition network powered by gated recurrent units (GRU). This architecture ensures stable and accurate gesture recognition even during interaction with physical objects. To evaluate its quantitative performance, we collected a custom dataset based on a predefined gesture set, achieving 93.0% accuracy in 5-fold cross-validation. The complete system implemented on Microsoft HoloLens 2 operates at a real-time framerate, and we further analyze the latency of each step in our framework. Through this interaction paradigm, users can experience immersive and intuitive AR in everyday environments with minimal disruption to natural action behavior. Our projects are available at https://github.com/kaist-uvrlab/UnifiedHOInteraction.

IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW) 2026

We present a VR research demo for practical telepresence with photorealistic 3D Gaussian Splatting (3DGS) avatars under HMD-only sensing. The demo consists of two parts that provide complementary experiences in a single session: a face part, where HMD-provided blendshape signals drive an expressive 3DGS head avatar rendered stereoscopically on the client, and a full-body part, where a user-specific 3DGS full-body avatar is created from a single image and driven using HMD signals. During the demo, participants first capture a single image for full-body avatar creation, experience facial control using a pre-built third-person head avatar while the full-body avatar is generated, and then control their personalized full-body avatar in VR. By enabling both facial and full-body controllability and contrasting two rendering-responsibility designs, client-side direct 3D rendering and server-side stereoscopic image rendering/streaming-the demo offers practical insights for designing responsive, realistic VR telepresence systems under HMD-only sensing.

IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW) 2026

We present a real-time AR system that enables interaction with physical cuboid- and cylinder-shaped objects by representing them as compact superquadrics. Our approach uses Instant Surface Normal (ISN) maps-color images encoding per-pixel geometry-to detect and segment objects based on shape rather than appearance. Superquadric parameters are extracted sequentially, allowing efficient and updatable 3D modeling in real time. Experimental results show accurate detection, consistent segmentation, and reconstruction quality comparable to existing methods, while achieving over 5× faster inference. This system allows physical objects to be seamlessly integrated as interactive elements in dynamic AR environments.

IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW) 2026

We propose VR Zen Garden, a virtual reality (VR) application designed to support stress relief and meditation through interactive spatial soundscapes. The system enables users to create personalized auditory environments by combining two key interactions: Object Placement, which spatializes sound-emitting objects, and Sand Pattern Raking, which modulates sound properties through visual patterns. These inputs are processed by the Sound Spatializer and the Metaphoric Sound Mixer, allowing users to intuitively shape immersive 3D soundscapes in real time. VR Zen Garden offers an engaging and embodied approach to mindfulness, highlighting the potential of sound-driven design in immersive wellness applications.

IEEE Transactions on Visualization and Computer Graphics (Proc. IEEE VR'26)Top 10%

We present VRGaussianAvatar, an integrated system that enables real-time full-body 3D Gaussian Splatting (3DGS) avatars in virtual reality using only head-mounted display (HMD) tracking signals. The system adopts a parallel pipeline with a VR Frontend and a GA Backend. The VR Frontend uses inverse kinematics to estimate full-body pose and streams the resulting pose along with stereo camera parameters to the backend. The GA Backend stereoscopically renders a 3DGS avatar reconstructed from a single image. To improve stereo rendering efficiency, we introduce Binocular Batching, which jointly processes left and right eye views in a single batched pass to reduce redundant computation and support high-resolution VR displays. We evaluate VRGaussianAvatar with quantitative performance tests and a within-subject user study against image- and video-based mesh avatar baselines. Results show that VRGaussianAvatar sustains interactive VR performance and yields higher perceived appearance similarity, embodiment, and plausibility. Project page and source code are available at https://vrgaussianavatar.github.io.

HCI Korea 2026

SIGGRAPH Asia 2025 XR

We propose a collaborative scene mood authoring system in a virtual reality (VR) environment, where multiple users design and control the multimodal feedback (i.e., audiovisual and haptic) of a shared scene using voice commands. To support this, the user’s command is processed by large language models (LLMs) with the scene metadata and mood presets to identify intended target objects and mood, and the system modifies the objects to match the desired atmospheric expression. To cover a wide range of affective moods in scene mood authoring, we devised four distinct mood presets that represent different combinations of arousal and valence across the affective space. Each preset is expressed through a set of labeled visual, auditory, and haptic feedback elements, enabling coherent and multimodal expression of affective states within the scene.

IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct) 2025

We present a spatial augmentation system designed for Augmented Reality (AR) exhibition experiences, focusing on the immersive reinterpretation of two-dimensional artworks. Existing AR exhibition applications primarily augment content related to the artwork, but fall short of its architectural elements, which are crucial for integrating the artwork into the viewer’s surrounding space and enabling immersive spatial alignment. This study proposes a spatial augmentation approach that takes an artwork image as input, generates floor patterns and architectural 3D models through 2D and 3D model generation pipelines, and semantically aligns and augments them in the physical space. This spatial augmentation intends to enhance viewer engagement and perceived presence by maintaining spatial coherence between virtual and real space. By activating architectural surfaces as part of the experience, the system repositions AR exhibitions as spatial narratives rather than conventional overlay-driven displays.

IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct) 2025

We propose a lifestyle quantification method for personalized augmented reality (AR) intervention in everyday life. Maintaining a healthy lifestyle accompanies extensive workforce and costs, because accurate health evaluation and tailored management is a prerequisite. However, research on assessing multiple dimensions of health and timely administration of personalized information has been lacking. To address these issues, first we deduced participant’s physical activity, nutrition, interpersonal relations, and stress management level through their mobile devices. Each participant’s week-long data were filtered by clinical features evidenced in high-level medical literature. Next, based on their health status, we presented personalized AR interventions in their simulated everyday environment. Statistical analysis and user experiment results verified the clinical capability of our method and its effect on health-promoting decision behaviors of participants. Through this feasibility study, we hope to contribute on tailored lifestyle management in our daily lives, for better health of all.

IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct) 2025

We present a fast and efficient method for transferring facial textures onto SMPL-X–based full-body avatars. Unlike conventional affine-transform methods that are slow and prone to visual artifacts, our method utilizes a barycentric UV conversion technique. Our approach precomputes the entire UV mapping into a single transformation matrix, enabling texture transfer in a single operation. This results in a speedup of over 7000x compared to the baseline, while also significantly improving the final texture quality by eliminating boundary artifacts. Through quantitative and qualitative evaluations, we demonstrate that our method offers a practical solution for personalization in immersive XR applications. The code is available online.1

IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct) 2025

We present a multimodal framework that uses few-shot language models to generate action scene graphs from raw video for structuring spatial experiences in extended reality (XR). The system integrates unsupervised action segmentation, multimodal captioning with a vision-language model (VLM) guided by speech and motion cues, and scene graph generation from captions using a large language model (LLM), both leveraging few-shot prompting. By combining visual, verbal, and motion inputs, the framework produces semantically rich and accurate representations of spatial experiences without requiring annotated data or model fine-tuning. The resulting outputs will support a range of downstream XR applications, including adaptive playback, intent-driven retrieval, and flexible content reconstruction.

IEEE International Symposium on Mixed and Augmented Reality (ISMAR) 2025Top-tier

As immersive virtual environments become more prevalent, avatars serve as critical social interfaces. This study explores how combinations of visual appearance, vocal characteristics, and informed identity influence users' initial impressions and partner preferences in four distinct XR scenarios: physical, intellectual, social, and romantic. A within-subject experiment with 40 participants assessed perceived warmth, competence, uncanniness, and selection preferences across diverse avatar configurations. Results indicate that vocal cues had a particularly strong impact on social perception, often shaping feelings of approachability and clarity in communication. While some cue alignments enhanced perceived social comfort and engagement, inconsistencies across gender-related cues occasionally led to increased perceptions of uncanniness, especially in emotionally sensitive contexts. These findings highlight the importance of designing avatars that thoughtfully adapt to different interaction contexts, supporting inclusive and responsive user experiences in social XR platforms.

IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct) 2025

This paper presents a comprehensive framework for standardizing avatar face representation in Extended Reality (XR)-based communication. It defines a consistent representation model for facial features and expressions to ensure high visual fidelity and expressive accuracy across diverse XR devices. The proposed framework supports interoperability across different systems and applications by defining core data structures that adapt to the hardware constraints and capabilities of different setups. A key focus is the real-time synchronization of facial expressions, even when users’ faces are occluded by XR devices, to enhance social presence and interaction quality. To address variability in user devices and network conditions, the framework introduces an adaptive control mechanism for real-time avatar rendering. This mechanism dynamically adjusts the visual complexity and expressiveness of avatars based on device performance and network bandwidth, ensuring consistent and responsive communication experiences in heterogeneous environments. Ultimately, this standardization effort aims to advance immersive communication by enabling lifelike, responsive avatars that reflect users’ expressions, thereby accelerating the adoption of XR technologies in both professional and social domains.

IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct) 2025

This paper proposes a mutual space representation to support remote collaboration in Mixed and Augmented Reality (MAR) environments. To achieve high immersion and coexistence in MAR remote collaboration, a mutual space based on heterogeneous physical spaces that can be commonly utilized by collaborating participants is required. Although previous studies have proposed various spatial matching and modifying methods to overcome spatial heterogeneity, their different representations have made their integrated use in real-world scenarios difficult. To overcome these limitations, this paper proposes a mutual space representation standard that includes the representation of physical space, matching space, and collaboration context required for mutual space generation and management. We expect that the proposed standards will serve as a foundation for seamlessly applying solutions accumulated in academia to industrial systems, thereby providing users with a more immersive and coexistent MAR remote collaboration experience.

IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct) 2025

We propose a system that provides personalized interpretation of physical paintings using an AR cursor in a mixed reality (MR) environment. In galleries, pointing at specific regions of complex paintings is often restricted, making it hard to ask targeted questions. While prior works have proposed interactive systems to assist visitor understanding, they typically offered generalized explanations that were not grounded in real, physically exhibited artworks. To address this gap, our approach allows users to intuitively drag across the surface of a painting using a hand-guided AR cursor to specify a Region of Interest. The cropped image, along with the user’s spoken question, is then sent to a Vision Language Model (VLM), which generates a personalized and context-aware explanation based on the selected area. This approach enables users to gain a more precise and meaningful understanding of the painting, guided by their specific interests.

IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct) 2025

We propose Round-Trip2Gesture, an interaction system that uses smartwatch IMU sensors to enable inplace gesture input free from the sensing FOV of devices. Current smartglasses input is limited by social unsuitability of voice commands and by the restricted sensing field-of-view inherent to hand tracking. Our system introduces a delimiter gesture based on specific wrist motions to achieve reliable interaction in real-world scenarios. Our system enables intuitive gesture interaction free from occulsion and physical constraints. We evaluated a set of 5 gestures with 8 participants. Four gestures showed over 90% accuracy in leave-one-subject-out cross-validation. We introduce the potential of the system through notification dismissal and video playback control applications, demonstrating seamless out-of-FOV interaction in Mixed Reality environments.

IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct) 2025

We propose a sequential context engineering framework for zero-shot recognition of procedural tasks in egocentric AR. While large vision-language models excel at zero-shot image classification, they lack temporal reasoning required for procedural understanding, and fully fine-tuning them degrades generalization capabilities. The proposed method preserves frozen CLIP encoders while introducing lightweight modules: an intra-action attention module for temporal aggregation and an inter-action context module that leverages previous predictions to generate context-aware prompts. Evaluated on EGTEA Gaze+ dataset, our approach shows promising results with notable improvements over baselines while maintaining parameter efficiency. This work presents a step toward enabling temporal reasoning for VLMs in resource-constrained AR systems without sacrificing zero-shot generalization.

IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct) 2025

We propose the Spatio-Temporal Mixed and Augmented Reality Experience Description (MAR-ED), a novel framework to standardize the representation of past events for interactive and adaptive playback in a user’s present physical space. While current spatial media technologies have primarily focused on capturing or replaying content as static assets, often disconnected from the viewer’s environment or offering limited interactivity, the means to describe an experience’s underlying semantic and interactive structure remains underexplored. We propose a descriptive framework called MARED based on three core primitives: 1) Event Primitives for semantic scene graph representation, 2) Keyframe Primitives for efficient and meaningful data access, and 3) Playback Primitives for user-driven adaptive interactive playback of recorded MAR experience. The proposed flowchart of the three-stage process of the proposed MAR-ED framework transforms a recorded experience into a unique adaptive MAR experience during playback, where its spatio-temporal structure dynamically conforms to a new environment and its narrative can be altered by live user input. Drawing on this framework, personal digital memories and recorded events can evolve beyond passive 2D/3D videos into immersive, spatially-integrated group experiences, opening new paradigms for training, cultural heritage, and interactive storytelling without requiring complex, per-user adaptive rendering.

Adjunct Proceedings of the 38th Annual ACM Symposium on User Interface Software and Technology

Figure 1: RealityCrafter creates an editable 3D scene driven by user speech from a single real-world image.Our system (a) takes a single image as input; (b) reconstructs it into a 3D scene; and (c) empowers users to author that space via voice commands and pointer interactions.

Proceedings of the 2025 CHI Conference on Human Factors in Computing SystemsTop-tier

This research investigates the use of hybrid user interfaces to enhance text readability in augmented reality (AR) by combining optical see-through head-mounted displays with smartphones. While this integration can improve information legibility, it may also introduce display switching side effects. The extent to which these side effects hinder user experience and when the benefits outweigh drawbacks remain unclear. To address this gap, we conducted an empirical study (N=24) to evaluate how hybrid user interfaces affect AR reading tasks across different content distances, which induce varying levels of display switching. Our findings show that hybrid user interfaces offer significant readability benefits compared to using the HMD only, reducing mental and physical demands when reading text linked to content at closer distances. However, as the distance between displays increases, the compensatory behaviors users adopt to manage increased switching costs negate these benefits, making hybrid user interfaces less effective. Based on these findings, we suggest (1) using smartphones as supplementary displays for text in reading-intensive tasks, (2) implementing adaptive display positioning to minimize switching overhead in such scenarios, and (3) adjusting the smartphone’s role based on content distance for less intensive reading tasks. These insights provide guidance for optimizing smartphone integration in hybrid interfaces and enhancing AR systems for reading applications.

IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW) 2025

Asynchronous communication is essential in diverse contexts such as remote meetings, education, and collaboration. However, traditional replay-based systems lack interactivity and often fail to preserve critical contextual cues. This paper introduces Avagent, a novel framework for asynchronous com munication in Virtual/Augmented Reality (VR/AR) environments that leverages agentized avatars to bridge temporal and contextual gaps between participants. Unlike typical agent-mediated systems, the framework utilizes users’ data to create Avagents that reflect their actual intent, emotional states, and behaviors. By accurately capturing and replicating both verbal and nonverbal behaviors, Avagents enable more interactive and context-rich communication. This approach facilitates dynamic interactions, enhances social presence, and ensures continuity across timelines. Avagents empower new users to engage interactively with prior discussions, fostering deeper understanding and seamless collaboration with previous users. Envisioning the benefits of Avagents as an engaging and context-rich solution for asynchronous communication, this paper outlines interaction scenarios, a work-in-progress prototype, and associated challenges.

IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW) 2025

We present a new approach using Virtual Reality to capture human-scene interaction data within 3D environments. Traditional methods required costly physical setups with real furniture, making it difficult to collect both interaction and 3D scene data comprehensively. Our solution implements a user-centric scene graph with a hierarchical structure that captures both user interactions and 3D scene elements. The resulting dataset contains real human interaction data, dynamic scene information, synthetic sensor data, and scene graph annotations. This work provides a foundation for developing Mixed Reality experiences using 3D scene graphs with human interaction data.

한국소프트웨어종합학술대회 (KSC 2025)

IEEE Transactions on Visualization and Computer GraphicsTop 10%

Immersive technologies such as virtual and augmented reality (VR/AR) allow remote users to meet and interact in a shared virtual space using embodied virtual avatars, creating a sense of co-presence. However, asynchronous communication-essential in many real-world contexts-remains underexplored in these environments. Traditional playback-based systems lack interactivity and often fail to preserve critical contextual cues necessary for effective asynchronous communication. In this paper, we introduce AVAGENTs, AI-powered virtual avatars that replicate users' verbal and nonverbal cues from recordings of past meetings. Avagents can interpret meeting context and generate appropriate responses to questions posed by asynchronous viewers. Through a user study (N = 30), we evaluated Avagents against a traditional playback method and a voice-based AI assistant across two asynchronous meeting scenarios: analytic reasoning and affective resonance. Results showed that Avagents enhance the asynchronous communication experience by increasing social presence, sense of belonging, emotional intimacy, and other user perceptions. We discuss the findings and their implications for designing effective AI-driven asynchronous communication tools in VR/AR environments.

IEEE Transactions on Visualization and Computer Graphics (Proc. IEEE VR'25)Top 10%

As avatars have evolved from simple digital representations into extensions of our identities, they offer unprecedented opportunities for self-expression and customization beyond the physical world limitations. While virtual platforms foster new forms of identity exploration, social norms still play a crucial role in defining what is considered appropriate in these environments. In this study, we surveyed 150 participants to investigate social norms surrounding avatar modifications, examining how perspectives, contexts, and personality traits influence attitudes toward appropriateness. Our findings reveal that avatar modifications are generally viewed as more appropriate when considered from a partner's perspective, especially for changeable attributes. However, these modifications are perceived as less acceptable in professional settings such as workplaces. Additionally, individuals with high self-monitoring tendencies tend to be more resistant to changes, while those scoring higher on Machiavellianism are more accepting of changes, particularly regarding unchangeable attributes and emotional expressions. These findings provide valuable insights for platform developers and designers, highlighting the importance of implementing context-aware customization options that balance core identity elements with personality-driven preferences, thereby enhancing user experiences while respecting social norms.

IEEE Computer Graphics and Applications

We introduce the concept of a meta-object, a next-generation virtual object that inherits the form, properties, and functions of its real-world counterpart, enabling seamless synchronization, interaction, and sharing between the physical and virtual worlds. While plenty of today's virtual objects provide some sensory feedback and dynamic behavior, meta-objects fully integrate interactive and multisensory features within a structured data framework to enable real-time immersive experiences in a postmetaverse intelligent simulation platform. Three key components underpin the utilization of meta-objects in the postmetaverse: property-embedded modeling for physical and action realism, adaptive multisensory feedback tailored to user interactions, and a scene graph-based intelligence simulation platform for scalable and efficient ecosystem integration. By leveraging meta-objects through wearable augmented reality/virtual reality devices, the postmetaverse facilitates seamless interactions that transcend spatial and temporal barriers, paving the way for a transformative reality-virtuality convergence.

GMCV Workshop at CVPR'25 (CVPRW)

Generating high-quality 3D models from 2D sketches is a challenging task due to the inherent ambiguity and sparsity of sketch data. In this paper, we present S3D, a novel framework that converts simple hand-drawn sketches into detailed 3D models. Our method utilizes a U-Net-based encoder-decoder architecture to convert sketches into face segmentation masks, which are then used to generate a 3D representation that can be rendered from novel views. To ensure robust consistency between the sketch domain and the 3D output, we introduce a novel style-alignment loss that aligns the U-Net bottleneck features with the initial encoder outputs of the 3D generation module, significantly enhancing reconstruction fidelity. To further enhance the network's robustness, we apply augmentation techniques to the sketch dataset. This streamlined framework demonstrates the effectiveness of S3D in generating high-quality 3D models from sketch inputs. The source code for this project is publicly available at https://github.com/hailsong/S3D.

IEEE International Symposium on Mixed and Augmented Reality (ISMAR) 2024Top-tier

We propose a novel crowd data-driven optimization approach for artwork placement in virtual exhibitions. With the emerging concept of Metaverse, a multitude of users can engage with content contemporaneously in virtual exhibitions. Yet, few studies have suggested a method to resolve crowd density concentration in multiuser Mixed Reality (MR) and Virtual Reality (VR) environments. In this study, our approach leverages crowd data engaged with artworks to predict optimal placement of artworks to distribute crowd density in virtual exhibitions, prior to exhibition planning. To investigate the requirement and validity of our approach, we conducted focus group interviews and an artwork relocation experiment as preliminary studies. In the generation of solution scenes for optimal placement, our optimizer adaptively scrutinizes placeable areas with considerations of crowd density distribution, scene rationality, and artwork similarity. Through a performance comparison analysis between optimization results, we confirmed that the optimizer successfully fulfilled the intended objectives with respect to the design considerations, resolving practical scenarios in exhibition planning.

IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct) 2024

We propose an algorithm that extracts the most suitable affordances, interaction targets, and corresponding coordinates adaptively from 3D models of various artifacts based on their functional context for efficient authoring of XR content with artifacts. Traditionally, authoring AR scenes to convey artifact context required one-to-one manual work. Our approach leverages a Large Language Model (LLM) to extract interaction types, positions, and subjects based on the artifact's name and usage context. This enables templated XR experience creation, replacing repetitive manual labor. Consequently, our system streamlines the XR authoring process, making it more efficient and scalable.

IEEE International Symposium on Mixed and Augmented Reality (ISMAR) 2024Top-tier

Understanding gender differences in facial and spatial recognition is crucial for enhancing avatar-mediated communication. However, there remains a gap in understanding how participant gender influences perceptions of avatar facial expressions and spatial dynamics in Mixed Reality communication. Therefore, our study investigates how avatar non-verbal cues interact with gender differences to affect user experience and understanding in MR environments. To examine these complex relationships, we conducted a user study comparing the effects of various avatar facial expressions (Full, Mouth-Only, and Emotion-based) and interpersonal distances (Closer vs. Farther) on facial animation realism and social presence, with a focus on gender-balanced participant groups. Our findings revealed that female participants were particularly sensitive to the avatar’s proximity and facial expressions, reporting significantly higher perceptions of facial animation realism, copresence, message understanding, and affective understanding at farther distances compared to male participants. They also perceived higher copresence and message understanding when exposed to emotion-based facial expressions, as opposed to a mouth-only condition-a distinction not observed among male participants. Based on our findings, we advocate for avatar design strategies that accommodate gender differences in perception and preference, potentially through customizable levels of expressiveness to cater to diverse user needs and contexts.

IEEE International Symposium on Mixed and Augmented Reality (ISMAR) 2024Top-tier

To enable remote Virtual Reality (VR) and Augmented Reality (AR) clients to collaborate as if they were in the same space during Mixed Reality (MR) telepresence, it is essential to overcome spatial heterogeneity and generate a unified shared collaborative environment by integrating remote spaces into a target host space. Especially when multiple remote users connect, a large shared space is necessary for people to maintain their personal space while collaborating, but the existing simple intersection method leads to the creation of narrow shared spaces as the number of remote spaces increases. To robustly align to the host space even as the number of remote spaces increases, we propose a spatial affordance-aware interactable subspace allocation algorithm. The key concept of our approach is to consider the perceivable and interactable areas separately, where every user views the same mutual space, but each remote user has a different interactable subspace, considering their location and spatial affordance. We conducted an evaluation with 900 space combinations, varying the number of remote spaces as two, four, and six, and results show our method outperformed in securing wide interactable mutual space and instantiating users compared to the other spatial matching methods. Our work enables multiple clients from diverse remote locations to access the AR host’s space, allowing them to interact directly with the table, wall, or floor by aligning their physical subspaces within a connected mutual space.

IEEE International Symposium on Mixed and Augmented Reality (ISMAR) 2024Top-tier

In avatar-mediated remote communication, avatars’ facial expressions can be dynamically adjusted according to each user’s computational and device constraints, highlighting the importance of varied expressions and their impact on user perception. However, there is a lack of research on how variations in avatar facial expressions, especially when simplified, influence user perception, particularly in terms of social presence. To address this, we examine the impact of various facial expression combinations on social presence in avatar-mediated communication scenarios, ranging from informative speeches to emotional conversations. Our approach involves prioritizing avatar facial blendshape combinations using two main approaches: (1) commonly activated expressions that reflect the active facial movements observed during casual conversations, and (2) emotion-based expressions derived from Facial Action Coding System (FACS). These combinations were compared against minimal baseline and full blendshape conditions through a comprehensive study involving 32 participants. Our findings reveal that emotion-based condition achieves comparable levels of social presence and communication quality to the full condition, in both informative speeches and emotional conversations. This highlights the effectiveness of prioritizing emotion-based expressions and adopting a streamlined approach to avatar facial control. By focusing on emotional expressions while optimizing resources, this approach shows potential for enhancing the avatar-mediated communication experience, accommodating the diverse users’ contexts.

IEEE International Symposium on Mixed and Augmented Reality (ISMAR) 2024Top-tier

We introduce “Whirling Interface,” a selection method for XR displays using bare-hand motion matching gestures as an input technique. We extend the motion matching input method, by introducing different input states to provide visual feedback and guidance to the users. Using the wrist joint as the primary input modality, our technique reduces user fatigue and improves performance while selecting small and distant targets. In a study with 16 participants, we compared the whirling interface with a standard ray casting method using hand gestures. The results demonstrate that the Whirling Interface consistently achieves high success rates, especially for distant targets, averaging 95.58% with a completion time of 5.58 seconds. Notably, it requires a smaller camera sensing field of view of only 21.45° horizontally and 24.7° vertically. Participants reported lower workloads on distant conditions and expressed a higher preference for the Whirling Interface in general. These findings suggest that the Whirling Interface could be a useful alternative input method for XR displays with a small camera sensing FOV or when interacting with small targets.

Adjunct Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology

Diminished reality (DR) removes virtual and real objects from the user’s view, decluttering and optimizing what the user sees through augmented/mixed reality. However, removing real objects create a potential safety concern because users may unintentionally bump into the diminished object. To address this issue, we apply characteristics of peripheral vision to DR and diminish objects when they are in the peripheral region and show the object when the user focuses on the object. We created 3 different object opacity control functions that use gaze information and evaluated them.

European Conference on Computer Vision (ECCV) 2024

Existing datasets for 3D hand-object interaction are limited either in the data cardinality, data variations in interaction scenarios, or the quality of annotations. In this work, we present a comprehensive new training dataset for hand-object interaction called HOGraspNet. It is the only real dataset that captures full grasp taxonomies, providing grasp annotation and wide intraclass variations. Using grasp taxonomies as atomic actions, their space and time combinatorial can represent complex hand activities around objects. We select 22 rigid objects from the YCB dataset and 8 other compound objects using shape and size taxonomies, ensuring coverage of all hand grasp configurations. The dataset includes diverse hand shapes from 99 participants aged 10 to 74, continuous video frames, and a 1.5M RGB-Depth of sparse frames with annotations. It offers labels for 3D hand and object meshes, 3D keypoints, contact maps, and grasp labels. Accurate hand and object 3D meshes are obtained by fitting the hand parametric model (MANO) and the hand implicit function (HALO) to multi-view RGBD frames, with the MoCap system only for objects. Note that HALO fitting does not require any parameter tuning, enabling scalability to the dataset’s size with comparable accuracy to MANO. We evaluate HOGraspNet on relevant tasks: grasp classification and 3D hand pose estimation. The result shows performance variations based on grasp type and object class, indicating the potential importance of the interaction space captured by our dataset. The provided data aims at learning universal shape priors or foundation models for 3D hand-object interaction. Our dataset and code are available at https://hograspnet2024.github.io/.

Springer Virtual Reality

We propose a robust 3D hand tracking system in various hand action environments, including hand-object interaction, which utilizes a single color image and a previous pose prediction as input. We observe that existing methods deterministically exploit temporal information in motion space, failing to address realistic diverse hand motions. Also, prior methods paid less attention to efficiency as well as robust performance, i.e., the balance issues between time and accuracy. The Temporally Enhanced Graph Convolutional Network (TE-GCN) utilizes a 2-stage framework to encode temporal information adaptively. The system establishes balance by adopting an adaptive GCN, which effectively learns the spatial dependency between hand mesh vertices. Furthermore, the system leverages the previous prediction by estimating the relevance across image features through the attention mechanism. The proposed method achieves state-of-the-art balanced performance on challenging benchmarks and demonstrates robust results on various hand motions in real scenes. Moreover, the hand tracking system is integrated into a recent HMD with an off-loading framework, achieving a real-time framerate while maintaining high performance. Our study improves the usability of a high-performance hand-tracking method, which can be generalized to other algorithms and contributes to the usage of HMD in everyday life. Our code with the HMD project will be available at https://github.com/UVR-WJCHO/TEGCN_on_Hololens2 .

Proceedings of the CHI Conference on Human Factors in Computing SystemsTop-tier

We investigate how path context, encompassing both comfort and attractiveness, shapes user experiences in outdoor locative storytelling using Augmented Reality (AR). Addressing a research gap that predominantly concentrates on indoor settings or narrative backdrops, our user-focused research delves into the interplay between perceived path context and locative AR storytelling on routes with diverse walkability levels. We examine the correlation and causation between narrative engagement, spatial presence, perceived workload, and perceived path context. Our findings show that on paths with reasonable path walkability, attractive elements positively influence the narrative experience. However, even in environments with assured narrative walkability, inappropriate safety elements can divert user attention to mobility, hindering the integration of real-world features into the narrative. These results carry significant implications for path creation in outdoor locative AR storytelling, underscoring the importance of ensuring comfort and maintaining a balance between comfort and attractiveness to enrich the outdoor AR storytelling experience.

Proceedings of the CHI Conference on Human Factors in Computing SystemsTop-tier

Augmented Reality (AR) is regarded as an innovative storytelling medium that presents novel experiences by layering a virtual narrative space over a real 3D space. However, understanding of how the virtual narrative space and the real space are connected with one another in the design of augmented narrative spaces has been limited. For this, we conducted a systematic literature review of 64 articles featuring AR storytelling applications and systems in HCI, AR, and MR research. We investigated how virtual narrative spaces have been paired, functionalized, placed, and registered in relation to the real spaces they target. Based on these connections, we identified eight dominant types of augmented narrative spaces that are primarily categorized by whether they virtually narrativize reality or realize the virtual narrative. We discuss our findings to propose design recommendations on how virtual-real connections can be incorporated into a more structured approach to AR storytelling.

Proceedings of the Augmented Humans International Conference 2024

We propose GestureMark, a novel input technique for target selection on XR glasses using smartwatch touch gestures as input. As XR glasses get smaller and lighter, their usage increases rapidly, leading to a higher demand for efficient shortcuts for everyday life. We explored the uses of gesture input on smartwatch touchscreen, including simple swipe, swipe combinations, and bezel-to-bezel (B2B) gesture as an input modality. Through an experiment with 16 participants, we found that while swipe gestures were efficient for four-choice selections, B2B was superior for 16-choice inputs. Feedback mechanisms did not enhance performance but reduced perceived workload. Our findings highlight the potential of integrating smartwatches as secondary input devices for XR glasses.

Proceedings of the Augmented Humans International Conference 2024

Assessing a patient’s social and environmental factors is crucial in medical treatment due to their impact on overall quality of life and treatment outcomes. However, current industrial solutions and research on augmented reality and virtual reality telehealth have primarily focused on training or physician collaboration, thereby overlooking the importance of assessing patients’ non-medical factors in the consulting environment. To address this issue, we propose a holistic patient assessment system in an extended reality (XR) teleconsultation environment, based on clinical evidence from the literature. By utilizing the digital twin (DT) of the patient’s environment and a physical examination assistance tool, healthcare professionals can effectively assess non-medical factors and enhance diagnosis accuracy. Through evaluations by current healthcare professionals, our system has been medically validated for its adequate integration of crucial factors in medicine and potential for real-world deployment. We believe that our system could envision a new line of research in future XR telehealth, particularly in the context of patient-centered and inclusive care.

IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW) 2024

This study proposes an experience graph, a Mixed Reality (MR) experience replaying method for recording and playing spatially adaptive interaction in a mixed reality environment. Many MR immersive interactions are experienced in a space different from the original recorded space, so it is important to adaptively experience the interaction in the local space. As a solution to the collision and mismatch during the interactions in local space, there is research on scene generation or adaptive object placement, but the change of interactions dependent on space and the resulting experience transformation were not considered. This study structures spatial information and dependent interactions into an experience graph and proposes a spatial adaptive replay method that minimizes interaction deformation. Recorded interactions are evaluated quantitatively through loss rate and transformation rate, played interactions are evaluated qualitatively through user experiments, and finally, indicators important to the user experience are reflected in the experience graph. Utilizing the results of this study will contribute to experiencing immersive spatial MR where users actively record and share spatial experiences beyond the limitations of time and space.

IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW) 2024

In avatar-mediated remote communication, reflecting one's facial expressions in avatars remains challenging but essential in facilitating effective communication and fostering social presence. From our prior studies, we underscored the significance of emotion-based facial expressions in avatar-mediated remote communications within eXtended Reality (XR). Our findings indicated that during emotional conversations as well as informative speeches, activating the emotion-based blendshapes resulted in a level of social presence and communication quality comparable to activating all facial blend-shapes. As we move forward, our research aims to investigate avatar facial expression considering XR contexts, with an emphasis on the challenges stemming from XR device constraints and the dynamic contexts of remote collaboration. We aspire to devise adaptive guidelines for avatar facial expression, leveraging emotional cues using XR devices. Additionally, we plan to systematically analyze the implications of remote collaboration's varying scenarios, formulating guidelines for adaptive avatar representation that account for different collaborative environments and scenarios. To enhance social presence and communication experiences among users, we are preparing for a series of user studies, aiming to validate the effectiveness of our approaches and prototype systems.

IEEE Conference on Virtual Reality and 3D User Interfaces (VR) 2024Top-tier

We propose Object Cluster Registration (OCR) using Geometric Spatial Affordance Graph (GSAG) to support user interaction with multiple objects in a shared space generated from two dissimilar rooms. Previous research on generating a shared virtual space from dissimilar real spaces has only reflected the information of individual objects and aimed at maximizing the area of the shared space. This led to limited interactions relying on the singular affordances of objects, neglecting to consider the usability and effectiveness of the generated shared spaces. The proposed OCR with GSAG, which considers the relationship between objects based on facing formation, extracts optimal object cluster pairs to align dissimilar rooms in generating shared virtual spaces. In an evaluation study involving 100 multi-cluster space pairs, applying OCR using GSAG showed greater effectiveness in preserving object correlations compared to cases where OCR was not used. Furthermore, the size of the shared space did not significantly differ between the two methods. This suggests that factoring in the relationship between objects does not compromise the objective of maximizing the shared virtual space. The proposed method is expected to serve as a foundation for generating shared virtual spaces that are more user-oriented and efficient by facilitating a wider range of collaborative activities for remote users in dissimilar real spaces with varied configurations.

IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW) 2024

Realistic avatars are fundamental for immersive experiences in Augmented Reality (AR) and Virtual Reality (VR) environments. In this work, we introduce a novel approach for avatar generation, combining 3D Gaussian Splatting with the parametric body model, SMPL. This methodology overcomes the inefficiencies of traditional image/video-based avatar creation, which is often slow and requires high computing resources. The integration of 3D Gaussian Splatting for representing human avatar offers realistic and real-time rendering for AR/VR applications. We also conducted preliminary tests to verify the quality of avatar representation using 3D Gaussian Splatting. These tests, displayed alongside outcomes from existing methods, demonstrate the potential of this research to significantly contribute to the creation of realistic avatars in the future. Additionally, several key discussions are presented, essential for developing and evaluating the system and providing valuable insights for future research.

IEEE International Symposium on Mixed and Augmented Reality (ISMAR) 2023Top-tier

We propose a novel space-rescaling technique for registering dissimilar physical-virtual spaces by utilizing the effects of adjusting physical space with redirected walking. Achieving a seamless and immersive Virtual Reality (VR) experience requires overcoming the spatial heterogeneities between the physical and virtual spaces and accurately aligning the VR environment with the user’s tracked physical space. However, existing space-matching algorithms that rely on one-to-one scale mapping are inadequate when dealing with highly dissimilar physical and virtual spaces, and redirected walking controllers could not utilize basic geometric information from physical space in the virtual space due to coordinate distortion. To address these issues, we apply relative translation gains to partitioned space grids based on the main interactable object’s edge, which enables space-adaptive modification effects of physical space without coordinate distortion. Our evaluation results demonstrate the effectiveness of our algorithm in aligning the main object’s edge, surface, and wall, as well as securing the largest registered area compared to alternative methods under all conditions. These findings can be used to create an immersive play area for VR content where users can receive passive feedback from the plane and edge in their physical environment.

IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct) 2023

In this paper, we delve into the influence of different facial blend-shape combinations on social presence and communication quality in avatar-mediated remote communication scenarios: informative speech and emotional conversation. The conducted user study (n = 32) underscores the crucial role of emotion-based blendshapes, suggesting that the emotionally essential blendshape combination outlines levels of social presence and communication quality. Our research highlights the potential of a streamlined approach for avatar facial control, facilitating emotional expressions and optimizing resources, fostering an avatar-mediated communication experience.

IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct) 2023

We propose the Holistic Quantified-Self (HQS), a sensor-driven user model for AR glass users. Existing research on context-aware AR interfaces typically relies on singular context attributes, resulting in an incomplete representation of user states. To address this gap, we propose an integrated user model for AR glass users, encompassing physical condition, cognitive-emotional status, social circumstances, and digital consumption behavior. Our system collects data from various devices, enabling simultaneous prediction of user states and tasks within the AR environment. We envision that this approach paves the way for future adaptive AR interfaces that can dynamically adjust virtual content based on the recognized user states and tasks.

IEEE International Symposium on Mixed and Augmented Reality (ISMAR) 2023Top-tier

We present a novel method for avatar body generation that cumulatively updates the texture and normal map in real-time. Multiple images or videos have been broadly adopted to create detailed 3D human models that capture more realistic user identities in both Augmented Reality (AR) and Virtual Reality (VR) environments. However, this approach has a higher spatiotemporal cost because it requires a complex camera setup and extensive computational resources. For lightweight reconstruction of personalized avatar bodies, we design a system that progressively captures the texture and normal values using a single RGBD camera to generate the widely-accepted 3D parametric body model, SMPL-X. Quantitatively, our system maintains real-time performance while delivering reconstruction quality comparable to the state-of-the-art method. Moreover, user studies reveal the benefits of real-time avatar creation and its applicability in various collaborative scenarios. By enabling the production of high-fidelity avatars at a lower cost, our method provides more general way to create personalized avatar in AR/VR applications, thereby fostering more expressive self-representation in the metaverse.

Proceedings of the 2023 CHI Conference on Human Factors in Computing SystemsTop-tier

We report on a qualitative study in which 22 participants created Augmented Reality (AR) stories for outdoor cultural heritage sites. As storytelling is a crucial strategy for AR content aimed at providing meaningful experiences, the emphasis has been on what storytelling does, rather than how it is done, the end user’s needs prioritized over the author’s. To address this imbalance, we identify how recurring patterns in the spatial trajectories and narrative compositions of AR stories for cultural heritage sites are linked to the author’s intent and creative process: While authors tend to bind story arcs tightly to confined trajectories for narrative delivery, the need for spatial exploration results in thematic content mapped loosely onto encompassing trajectories. Based on our analysis, we present design recommendations for site-specific AR storytelling tools that can support authors in delivering their intent while leveraging the placeness of cultural heritage sites as a creative resource.

Proceedings of the 2023 CHI Conference on Human Factors in Computing SystemsTop-tier

An omni-directional (360°) camera captures the entire viewing sphere surrounding its optical center. Such cameras are growing in use to create highly immersive content and viewing experiences. When such a camera is held by a user, the view includes the user’s hand grip, finger, body pose, face, and the surrounding environment, providing a complete understanding of the visual world and context around it. This capability opens up numerous possibilities for rich mobile input sensing. In OmniSense, we explore the broad input design space for mobile devices with a built-in omni-directional camera and broadly categorize them into three sensing pillars: i) near device ii) around device and iii) surrounding device. In addition we explore potential use cases and applications that leverage these sensing capabilities to solve user needs. Following this, we develop a working system to put these concepts into action, by leveraging these sensing capabilities to enable potential use cases and applications. We studied the system in a technical evaluation and a preliminary user study to gain initial feedback and insights. Collectively these techniques illustrate how a single, omni-purpose sensor on a mobile device affords many compelling ways to enable expressive input, while also affording a broad range of novel applications that improve user experience during mobile interaction.

IEEE Conference on Virtual Reality and 3D User Interfaces (VR) 2023Top-tier

The reading experience on current augmented reality (AR) head mounted displays (HMDs) is often impeded by the devices' low perceived resolution, translucency, and small field of view, especially in situations involving lengthy text. Although many researchers have proposed methods to resolve this issue, the inherent characteristics prevent these displays from delivering a readability on par with that of more traditional displays. As a solution, we explore the use of smartphones as assistive displays to AR HMDs. To validate the feasibility of our approach, we conducted a user study in which we compared a smartphone-assisted hybrid interface against using the HMD only for two different text lengths. The results demonstrate that the hybrid interface yields a lower task load regardless of the text length, although it does not improve task performance. Furthermore, the hybrid interface provides a better experience regarding user comfort, visual fatigue, and perceived readability. Based on these results, we claim that joining the spatial output capabilities of the HMD with the high-resolution display of the smartphone is a viable solution for improving the reading experience in AR.

IEEE Conference on Virtual Reality and 3D User Interfaces (VR) 2023Top-tier

Augmented reality (AR) helps users easily accept information when they are walking by providing virtual information in front of their eyes. However, it remains unclear how to present AR notifications considering the expected user reaction to interruption. Therefore, we investigated to confirm appropriate placement methods for each type by dividing it into notification types that are handled immediately (high) or that are performed later (low). We compared two coordinate systems (display-fixed and body-fixed) and three positions (top, right, and bottom) for the notification placement. We found significant effects of notification type and placement on how notifications are perceived during the AR notification experience. Using a display-fixed coordinate system responded faster for high notification types, whereas using a body-fixed coordinate system resulted in quick walking speed for low ones. As for the position, the high types had a higher notification performance at the bottom position, but the low types had enhanced walking performance at the right position. Based on the finding of our experiment, we suggest some recommendations for the future design of AR notification while walking.

IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW) 2023

We propose Memo:me, an AR sticky note with priority-based color transition and on-time reminders in smartphones. For the priority-based color transition, the user can choose by himself among three different colors, or the system changes the color automatically 10 minutes before the entered time. For the on-time reminder, Memo:me provides a visual notification and a sound at the designated time. We enabled users to create virtual notes on planes, or carry daily objects with the virtual notes attached. We expect that our system would benefit users to manage their tasks in a time-appropriate manner.

IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW) 2023

We propose a Speech-Pose integration method to overcome the limitation of existing body pose estimation. Unlike previous Speech-based gesture generation method, our proposal reflects the user's actual pose using a vision-based system and speech as a subsidiary. When the system detects the camera out of sight, we carry out a context-aware method analyzing the speech. The system determines the target pose and connects it with the last pose captured by the camera using the bounding box. Our system can be a robust solution for avatar-mediated remote collaboration which requires accurate gesture delivery such as VR remote yoga training.

IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct) 2022

We propose a system concept for Augmented Reality Head-Mounted Display users, which supports multitask viewing with multiple vir-tual workspaces anchored in the real-world space. Although people encounter multitasking necessities frequently, the native AR HMD and existing interfaces lack measures to visualize multiple sets of spatially-anchored information in parallel. The system separately vi-sualizes two different sets of spatially-anchored information, one on each AR HMD and smartphone, enabling side-by-side multitasking on AR HMD without applying heavy load on users. We implemented a proof-of-concept prototype that allows side-by-side viewing of the two different virtual workspaces. The proposed concept shows promises of multitasking on AR HMD, and future research will de-velop the system to be fully functional and verified with user studies.

IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct) 2022

We propose an AR store authoring system that enables the users to customize a physical space to promote spatial experience in mobile AR shopping. The current state of AR shopping research attempts to enhance the shopping experience by presenting virtual items onto the user's space. However, under the notion that the stores and products are inseparable in creating a holistic shopping experience, we set to make a shopping experience that encompasses both elements. Instead of simply augmenting a whole store, this study introduces a space adaptive AR store that divides a virtual store into multiple sections and places them onto physical space according to the labeled surfaces. This prototype mobile application intends to deliver a complete spatial shopping experience by allowing the users to customize the stores in their space and ultimately attach the store experience to the AR shopping system. This study aims to introduce new scalability of the AR technology application in remote shopping.

IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct) 2022

We propose DirectionQ, an interaction technique using a continuous mid-air hand input that shows directional visual cues on objects to enable fast and accurate multiple target selection in dense environments. The user can select various targets in a row by moving their hand consecutively in the same direction with the visual cues of targeted objects. DirectionQ involves assigning directional visual cues to each object in the user's viewpoint, detecting the trajectory changes of hand movement, and inferring intended targets with directions of the segmented hand movement. Unlike traditional approaches for multiple target selection, DirectionQ does not need additional steps to disambiguate, providing less time for multiple target selection. While distant, small, or occluded objects often bring accuracy or time-consuming problems, DirectionQ shows similar performance regardless of target conditions. To prove the effectiveness of our approach, we conducted two controlled experiments comparing our system with two existing techniques, RayCasting and EyeGaze. The results show that DirectionQ is faster and more accurate than two techniques with a lower cognitive load on multiple target selection.

IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct) 2022

This study explored how avatar face Level of Detail (LOD) and task types affect users' social presence and interpersonal communication in Augmented Reality remote collaboration. In AR remote collaboration, users are represented as avatars, and an actual user's appearance inevitably changes. While certain situations require an avatar that resembles a user, it is challenging to reconstruct everything realistically due to its high cost. Since facial parts show relative significance compared to body parts, we proposed controlling face LOD as an approach to lowering the cost while maintaining social presence in AR environments. Our results showed that while Mid-LOD maintains social presence but reduces costs compared to High-LOD, Low-LOD negatively impacts social presence mainly due to emotional understanding. Based on our findings, we highlight the importance of 1) considering facial expressions and emotional understanding in managing face LOD, 2) figuring out the collaboration context with different effects of tasks and communication behaviors, and 3) providing additional communication cues for maintaining social presence while lowering the cost.

IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct) 2022

Augmented reality head-mounted displays (AR HMD) enable visual notifications directly into the user's field of view (FoV) to perform dual-task. However, research on the efficient presentation of notifications when communicating with a virtual avatar in augmented reality (AR) situation has been insufficient. For efficient AR notification presentations, it is necessary to understand where to display notifications without interfering with virtual avatar communication. We investigated two coordinate systems (world-fixed and display-fixed) and eight positions for AR notifications when conversing with the virtual avatar. Results showed that the top-left position increased response time and task load. Our study contributes as a cornerstone to designing AR notifications in communication with virtual avatars.

IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct) 2022

Recent advances in digital heritage have led to participatory interpretation that enables the creation of content provided by individuals as well as content from official cultural institutions. Although researchers have developed systems supporting participatory interpretation, little is known about how people experience participatory interpretation that is socially generated. To investigate this, we developed a mobile Virtual Reality application called MyInterpret and conducted a user study with 17 participants. The study results showed that user-generated interpretation significantly improving user experience, participants created and perceived user-generated interpretation differently from the official interpretation, and social features were used for engaging with user-generated content. We outline the implications of our work for curators and designers, and discuss how they may benefit from considering ‘participatory interpretation’ in the future.

IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct) 2022

This study proposed a Mixed Reality (MR) mutual space gener-ation method that aligns the Virtual Reality (VR) clients' spaces around the Augmented Reality (AR) host space's reference object and adjusts the virtual space's scale with translation-based Redi-rected Walking (RDW). Although there were methods to create shared spaces by finding optimal transform or each space to max-imize the semantically aligned area, there was a lack of research on utilizing the effect of rescaling a VR client's space with the RDW technique. We leveraged Relative Translation Gains (RTGs), a length-based RDW technique, to maximize the semantic information matching ratio between the AR host space and the VR clients' spaces by rescaling the client's virtual space within a suitable threshold range according to the spatial configuration. Our pilot study results show that the semantic information matching rate between the AR and the VR space increased compared to the existing one-to-one scale matching method while maintaining a unified coordinate sys-tem. Our findings can be used to create a more expansive mutual space for asymmetric remote collaboration even if remote spaces increase and challenging to secure a sufficient shared area.

IEEE International Symposium on Mixed and Augmented Reality (ISMAR) 2022Top-tier

This paper evaluates factors of social presence during a dynamic remote collaboration task in a technologically asymmetric Mixed Reality (MR) setting for two spatial layouts. While active movement during MR remote collaboration is afforded by how the shared 3D space is mediated and configured, studies investigating the impact of these conditions on user experience have been scarce. In a between-group study $(\mathrm{n}=48)$, a host user in Augmented Reality (AR) and a remote user in Virtual Reality (VR), both wearing Head Mounted Displays (HMDs), simultaneously moved around the shared space to find and assemble parts of a Mars exploration rover together, one group in a Peripheral layout and the other in a Scattered layout with disparate levels of spatial affordance. Results show that while VR facilitates higher co-presence and spatial presence than AR through HMDs, the Peripheral layout enables users to pay more attention to one another than the Scattered. We analyze the results and derive implications aimed at bridging the AR-VR gap in social presence for dynamic MR remote collaboration through the adaptive placement of virtual content in shared spaces.

IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct) 2022

We present a user interface method called TranSlider, designed for an AR collaboration with the concept of customized visualization of the user-selected augmented content. In AR, virtual content is augmented on top of the real world, so collaborating users are inevitably exposed to the diverse-overlapped information layers. Because the human recognition system has limited capacity, this overloaded information may burden the user to understand fully, and it is also challenging to manipulate among occluded objects. To solve this issue, we designed the TranSlider that allows users to control the presence of augmented content being transparent or highlighted based on the user interaction. The implemented prototype includes UI components, a manipulation function, and a network-connected collaboration module, and its applicability was verified through a preliminary interview. We expect that it would effectively support general AR collaborations and those with specific purposes, such as designing complex, hierarchical, or made up of many partial assets.

IEEE Conference on Virtual Reality and 3D User Interfaces (VR) 2022Top-tier

This paper investigates how the size of virtual space and objects within it affect the threshold range of relative translation gains, a Redirected Walking (RDW) technique that scales the user’s movement in virtual space in different ratios for the width and depth. While previous studies assert that a virtual room’s size affects relative translation gain thresholds on account of the virtual horizon’s location, additional research is needed to explore this assumption through a structured approach to visual perception in Virtual Reality (VR). We estimate the relative translation gain thresholds in six spatial conditions configured by three room sizes and the presence of virtual objects (3 × 2), which were set according to differing Angles of Declination (AoDs) between eye-gaze and the forward-gaze. Results show that both size and virtual objects significantly affect the threshold range, it being greater in the large-sized condition and furnished condition. This indicates that the effect of relative translation gains can be further increased by constructing a perceived virtual movable space that is even larger than the adjusted virtual movable space and placing objects in it. Our study can be applied to adjust virtual spaces in synchronizing heterogeneous spaces without coordinate distortion where real and virtual objects can be leveraged to create realistic mutual spaces.

IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW) 2022

Providing additional information in the proper position of augmented reality (AR) head-mounted display (HMD) can help increase AR performance and usability for dual-task. Therefore, our study inves-tigated how to place notifications for the dual-task to address this. We compared eight display positions and two tasks (single and dual tasks) to identify the appropriate area for displaying notifications. We confirmed that the middle-right reduces response time and task load. In contrast, the top-left is the location, which should avoid providing any notification in AR dual-task. Our study contributes to designing AR notifications on HMDs to enhance everyday AR experiences.

IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct) 2021

In remote collaboration using Augmented Reality (AR), speech and gesture are major communication methods for the general public. However, the Deaf and Hard-of-Hearing (DHH) population cannot join in the communication due to the absence of a sign language interface which is their primary language. Recent works have tried to augment spoken language with sign language animations or captions, but the research to convey sign language with spoken language is still very limited. In this paper, we propose a novel multi-modal communication system that integrates sign language translation, speech recognition, and shared object manipulation in the mobile AR environment. Though the system is currently under development, we demonstrated a rapid prototype of the telemedicine app leveraging the video prototyping method to integrate the system modules. We performed preliminary interviews about our approach with DHH users, a sign language interpreter, and a physician. We discuss the insights into the future design of the DHH communication support in the AR collaboration system. This study has a socio-cultural, economic impact on the DHH population as a barrier-free design of a remote collaboration system in a practical scenario. Another contribution of this work is that we suggested a novel user-centered system for DHH users in AR by integrating the existing technologies.

IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct) 2021

In this poster, we present a digital twin-based mixed reality system for remote collaboration with the size-scaling of the user and the space. The proposed system supports collaboration between an AR host user and a VR remote user by sharing a 3D digital twin of the AR host user. To enhance the coarse authoring of a shared digital twin environment, we provide a size scaling of the digital twin environment with the world-in-miniature view. Also, we enable scaling the size of the VR user’s avatar to enhance both coarse (size-up) and fine-grained (size-down) authoring of the digital twin environment. We describe the system setup, input methods, and interaction methods for scaling space and user.

Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems

In this work, we explore a new sensing technique for smart eyewear equipped with Electrooculography (EOG) sensors. We repurpose the EOG sensors embedded in a JINS MEME smart eyewear, originally designed to detect eye movement, to detect midair hand gestures. We also explore the potential of sensing human proximity, rubbing action and to differentiate materials and objects using this sensor. This new found sensing capabilities enable a various types of novel input and interaction scenarios for such wearable eyewear device, whether it is worn on body or resting on a desk.

Proceedings of the 2021 CHI Conference on Human Factors in Computing SystemsTop-tier

Space-adaptive algorithms aim to effectively align the virtual with the real to provide immersive user experiences for Augmented Reality(AR) content across various physical spaces. While such measures are reliant on real spatial features, efforts to understand those features from the user’s perspective and reflect them in designing adaptive augmented spaces have been lacking. For this, we compared factors of narrative experience in six spatial conditions during the gameplay of Fragments, a space-adaptive AR detective game. Configured by size and furniture layout, each condition afforded disparate degrees of traversability and visibility. Results show that whereas centered furniture clusters are suitable for higher presence in sufficiently large rooms, the same layout leads to lower narrative engagement. Based on our findings, we suggest guidelines that can enhance the effects of space adaptivity by considering how users perceive and navigate augmented space generated from different physical environments.

IEEE Virtual Reality and 3D User Interfaces (VR) 2021

We propose the concept of relative translation gains, a novel Redirected Walking (RDW) method to create a mutual movable space between the Augmented Reality (AR) host's reference space and the Virtual Reality (VR) client's space. Previous RDW methods have focused on maximizing the movable space at the expense of aligning the coordinates between the AR and VR side, and could only be applied to collaborative scenarios involving sequential tasks. Our method solves these problems by adjusting the remote client's walking speed for each axis of a VR space to modify the movable area without coordinate distortion. We estimate the relative translation gain threshold, defined as the extent to which the walking speed can be altered without creating a perceived difference in distance. In order to reflect features of the reference space in generating the mutual space, we then examine how changing its size affects the threshold value. Our study showed that for remote clients connected to the larger reference space, relative translation gains can be increased to utilize a VR space bigger than their real space. Our method can be applied to create optimal mutual spaces for a wider variety of asymmetric Mixed Reality (MR) remote collaboration systems.

IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW) 2021

We present a digital twin-based Mixed Reality platform for arts curators. Despite the rising presence of exhibitions in varying digital formats, the virtual engagement of arts curators has been nearly disabled. To replicate and enhance the professional capability of arts curators in the virtual realm, our system integrates a digital twin art space and 3D authoring techniques, aided with spatial and semantic metadata. Our user evaluation proves the system's usability, and its capacity to support curatorial activities. With this, we aim to provide a groundwork to similar systems that also extend arts curator's creativity and outreach beyond time and space.

IEEE Virtual Reality and 3D User Interfaces (VR) 2021

Most research on providing location-based content in 3D interactive virtual reality has been limited to social media content. Few studies have suggested how to represent the video clip of movies or TV shows in virtual reality. This paper investigates a video content representation method to provide a hyper-reality experience of the narrative world in virtual reality. We reflect the time and place settings of the video content in virtual reality and have participants watch the video in four different virtual reality environments. We reveal that reflecting the story's environment settings to the virtual reality environment significantly improves the spatial presence and narratives engagement. We also confirm a positive correlation between spatial presence and narrative engagement, including subscales such as emotional engagement and narrative presence. Based on the study results, we discuss how to provide the hyper-reality experience in content-adaptive virtual reality.

IEEE International Conference on Consumer Electronics (ICCE) 2021

This paper shows a structure and implementation of an Urban Digital Twin (UDT), simulating virtual data productions using Virtual IoT (VI) to design services for citizens. When designing smart city services with UDT systems, developing the system without data support from real IoTs can be created without knowing the scope and limitations of the IoTs. We simulated the services by creating VI system, which can be introduced in the future based on the meaningful data sets of existing Real IoT sensors. This paper focuses on the concept of the proposed system, GeoVCM, the components that make up the system.

IEEE International Symposium on Mixed and Augmented Reality (ISMAR) 2020Top-tier

Previous methods successfully estimated 3D hand poses from unblurred depth images with slow and smooth hand motions. However, the performance drops when the depth images are contaminated by motion blur due to fast hand motion. In this paper, we exploit an infrared (IR) image input, which is weakly blurred under fast hand motion. The proposed method is based on domain transfer learning from depth to infrared images. Note we do not have IR images with hand skeletons, thus proposing self-supervision rather than direct supervision using the skeleton labels. We train a Hand Image Generator (HIG) and two Hand Pose Estimators (HPEs) on paired depth and infrared images via self-supervision using a consistency loss, guided by an existing HPE trained on paired depth and hand skeleton entries. The IR-based HPE is then refined on the weakly blurred infrared images. The qualitative and quantitative experiments demonstrate that the proposed method accurately estimates 3D hand poses under motion blur by fast hand motion, while existing depth-based methods fail. Our solution therefore supports fast 3D manipulation of virtual objects for augmented reality applications. Our model and dataset are publicly available for future research.1

IEEE International Symposium on Mixed and Augmented Reality (ISMAR) 2020Top-tier

We propose a 3D hand tracking system using bare-hand depth inpainting from an RGB-depth image for a hand interacting with an object. The effectiveness of most existing hand-object tracking methods is impeded by the insufficiency of data, which do not include hand data occluded by the object, and their reliance on the information inferred from assuming the specific object type. We generate a sufficiently accurate bare-hand depth image from a hand interacting with an object using a conditional generative adversarial network, which is trained using the synthesized 2D silhouettes of the object to learn the morphology of the hand. We evaluate the proposed approach using a hierarchical particle filter-based hand tracker and prove that our approach utilizing the bare-hand tracker in the hand-object interaction dataset achieve state-of-the-art performance. The generalization of our work will enable visual-tactile interaction that is more natural in various wearable augmented reality applications.

IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct) 2020

Using an augmented reality head-mounted display(HMD) to extend the display of smartphone could open up new user interface solutions, benefiting users with an increased screen real-estate and visualizations that leverage each device's capabilities. Some previous works have explored the viability of extended displays, but knowledge regarding considerable design factors and constraints for such displays is still very limited. In our work, we conducted an exploratory study to investigate how different properties of augmented content in extended displays affect the user's task performance and subjective workload in an image comparison task. Through an experiment with 24 participants, we compared four augmented content placements (top, right, left, and bottom) and two augmented content sizes (small and large). The study results demonstrated that there are significant effects of both placement and size of augmented content on task performance and subjective workload. Based on the findings of our study, we propose some recommendations for the future design of extended displays.

IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct) 2020

Information in augmented reality (AR) consists of virtual and real contexts and is provided to the AR environment at different distances for each user. Therefore, it is important to understand how the integration of two different information influences the user's AR experience. However, little has been studied regarding the complexity and viewing distance of a real space. Our study investigates how the complexity of the physical environment and viewing distance influence workload and performance in a visual search task in an AR environment. We conducted an experiment in which participants performed conjunction search under three different levels of background complexity, at both near (1.5m) and far (3m) distances. The results indicated that as the complexity of the background increased, the users' performance time and workload were negatively impacted. In addition, when the distance between the user and the background was greater, search time increased. From the results of the study, we derived some recommendations for the design of AR interfaces. Our research contributes to the design of interfaces by demonstrating the necessity to consider the complexity of background and viewing distance.

IEEE International Symposium on Mixed and Augmented Reality (ISMAR) 2020Top-tier

This study investigates the effects of a virtual hand representation on the user experience including social presence during hand-based 3D remote collaboration. Although a remote hand appearance is a critical parts of a hand-based telepresence, it has been rarely studied in comparison to studies on the self-embodiment of virtual hands in a 3D environment. Thus, we conducted a user study comparing the three virtual hands models (Skeleton, Low Polygon and Realistic) while performing a remote collaborative task based on the American Sign Language (ASL) using both Augmented Reality (AR) and Virtual Reality (VR) environments. We found that the realistic type was perceived as the most sense of being together, human-like, and trustable representation. The low polygon model could also convey a clear sign and moderate level of social presence. Although the system was configured asymmetrically in AR and VR, little difference in perception was found except for the participant's mental load and message understanding. We then discuss the results and suggest design implications for future hand-based 3D telepresence systems.

Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems

This paper reports that the time-domain accuracy of bare-hand interactions in HMD-based Augmented Reality can be improved by using finger contact: touching a finger with another or tapping one's own hand. The activation of input can be precisely defined by the moment of finger contact, allowing the user to perform the input precisely at the desired moment. Finger contact is better suited to the user's mental model, and natural tactile feedback from the fingertip also benefits the user with the self-perception of the input. The experimental results revealed that using finger contact is the preferred method of input that increases the time-domain accuracy and enables the user to be aware of the moment the input is activated.

Proceedings of the Augmented Humans International Conference

Cheek kissing is a common greeting in many countries around the world. Many parameters are involved when performing the kiss, such as which side to begin the kiss on and how many times the kiss is performed. These parameters can be used to infer one's social and physical context. In this paper, we present KissGlass, a system that leverages off-the-shelf smart glasses to recognize different kinds of cheek kissing gestures. Using a dataset we collected with 5 participants performing 10 gestures, our system obtains 83.0% accuracy in 10-fold cross validation and 74.33% accuracy in a leave-one-user-out user independent evaluation.

Proceedings of the Augmented Humans International Conference

WristLens is a system for surface interaction from wrist-worn wearable devices such as smartwatches and fitness trackers. It enables eyes-free, single-handed gestures on surfaces, using an optical motion sensor embedded in a wrist-strap. This allows the user to leverage any proximate surface, including their own body, for input and interaction. An experimental study was conducted to measure the performance of gesture interaction on three different body parts. Our results show that directional gestures are accurately recognized but less so for shape gestures. Finally, we explore the interaction design space enabled by WristLens, and demonstrate novel use cases and applications, such as on-body interaction, bimanual interaction, cursor control and 3D measurement.

IEEE International Symposium on Mixed and Augmented Reality (ISMAR) 2019Top-tier

One of the main challenges in creating narrative-driven Augmented Reality (AR) content for Head Mounted Displays (HMDs) is to make them equally accessible and enjoyable in different types of indoor environments. However, little has been studied in regards to whether such content can indeed provide similar, if not the same, levels of experience across different spaces. To gain more understanding towards this issue, we examine the effect of room size and furniture on the player experience of Fragments, a space-adaptive, indoor AR crime-solving game created for the Microsoft HoloLens. The study compares factors of player experience in four types of spatial conditions: (1) Large Room - Fully Furnished; (2) Large Room - Scarcely Furnished; (3) Small Room - Fully Furnished; and (4) Small Room - Scarcely Furnished. Our results show that while large spaces facilitate a higher sense of presence and narrative engagement, fully-furnished rooms raise perceived workload. Based on our findings, we propose design suggestions that can support narrative-driven, space-adaptive indoor HMD-based AR content in delivering optimal experiences for various types of rooms.

IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct) 2019

Augmented reality is a technology which extends measureless contents in the virtual space and makes actual objects not limited in the physical properties. As the most popular cultural media among the campus, the poster have its shortcoming of offering restricted information. In this paper, we introduce a responsive poster-reading system which generates diverse contents step by step with body-based and hand-based interactions, and a design guideline which applies for corresponding posters. The proposed method is able to provide users with a user-friendly, immersive and intuitive experience.

Proceedings of the 23rd International Symposium on Wearable Computers

SelfSync enables rapid, robust initiation of a gesture interface using synchronized movement of different body parts. SelfSync is the gestural equivalent of a hotword such as OK-Google in a speech interface and is enabled by the increasing trend where a user wears two or more wearables, such as a smartwatch, wireless earbuds, or a smartphone. In a user study comparing five potential SelfSync gestures in isolation, our system averages 96%, 98% and 88% for user dependent, user adapted, and user independent accuracy, respectively. For when the user has a phone in a pocket and a smart-watch, we suggest twisting the hand about the wrist while moving the leg with the phone in synchrony left and right. When the user has a head worn device and a smartwatch, we suggest twisting the hand while twisting the head left and right.

Proceedings of the 2019 CHI Conference on Human Factors in Computing SystemsTop-tier

Many researchers have studied various visual communication cues (e.g. pointer, sketching, and hand gesture) in Mixed Reality remote collaboration systems for real-world tasks. However, the effect of combining them has not been so well explored. We studied the effect of these cues in four combinations: hand only, hand + pointer, hand + sketch, and hand + pointer + sketch, with three problem tasks: Lego, Tangram, and Origami. The study results showed that the participants completed the task significantly faster and felt a significantly higher level of usability when the sketch cue is added to the hand gesture cue, but not with adding the pointer cue. Participants also preferred the combinations including hand and sketch cues over the other combinations. However, using additional cues (pointer or sketch) increased the perceived mental effort and did not improve the feeling of co-presence. We discuss the implications of these results and future research directions.

IEEE Conference on Virtual Reality and 3D User Interfaces (VR) 2019Top-tier

We present a novel view synthesis method that allows users to experience a large-scale Six-Degree-of-Freedom (6-DOF) virtual environment. Our main contributions are the construction of a large-scale 6-DOF virtual environment using multiple 360 images as well as synthesis of a scene from novel viewpoints. Novel view synthesis from a single 360 image can give free viewpoint experience with full 6-DOF of head motion to players, but the moveable space is limited within a context of the image. We propose a novel view synthesis process that references multiple 360 images via reconstructing a large-scale real world based virtual data map and perform a weighted blending for interpolating multiple novel view images. Our results show that our approach provides a wider area of virtual environment as well as a smooth transition between each reference 360 images.

IEEE Conference on Virtual Reality and 3D User Interfaces (VR) 2019Top-tier

This paper investigates the effect of avatar appearance on Social Presence and users' perception in an Augmented Reality (AR) telepresence system. Despite the development of various commercial 3D telepresence systems, there has been little evaluation and discussions about the appearance of the collaborator's avatars. We conducted two user studies comparing the effect of avatar appearances with three levels of body part visibility (head & hands, upper body, and whole body) and two different character styles (realistic and cartoon-like) on Social Presence while performing two different remote collaboration tasks. We found that a realistic whole body avatar was perceived as being the best for remote collaboration, but an upper body or cartoon style could be considered as a substitute depending on the collaboration context. We discuss these results and suggest guidelines for designing future avatar-mediated AR remote collaboration systems.

IEEE International Conference on Consumer Electronics (ICCE) 2019

We propose the system that provides an immersive experience by automatically extending the video content of the TV to the wall. Though it is possible to manually create the appropriate content that works on the existing TV screen, the cost is too expensive to create. The proposed deep neural network uses the center of the video as input and the outside area as output. The trained network automatically generates the extrapolated video according to the input video and a projector projects the resulting content on the living room wall around the TV thereby improving the immersive feeling. The proposed system is expected to pave a new road to the home appliance industry by transforming the living room into the new immersive experience platform.

Proceedings of the 21st International Conference on Human-Computer Interaction with Mobile Devices and Services

To better explore the incorporation of pointing and gesturing into ubiquitous computing, we introduce WRIST, an interaction and sensing technique that leverages the dexterity of human wrist motion. WRIST employs a sensor fusion approach which combines inertial measurement unit (IMU) data from a smartwatch and a smart ring. The relative orientation difference of the two devices is measured as the wrist rotation that is independent from arm rotation, which is also position and orientation invariant. Employing our test hardware, we demonstrate that WRIST affords and enables a number of novel yet simplistic interaction techniques, such as (i) macro-micro pointing without explicit mode switching and (ii) wrist gesture recognition when the hand is held in different orientations (e.g., raised or lowered). We report on two studies to evaluate the proposed techniques and we present a set of applications that demonstrate the benefits of WRIST. We conclude with a discussion of the limitations and highlight possible future pathways for research in pointing and gesturing with wearable devices.

3rd Digital Heritage International Congress (DigitalHERITAGE) held jointly with 2018 24th International Conference on Virtual Systems & Multimedia (VSMM 2018)

Conventional approaches to a virtual heritage site that provide interpretations through curated content allow visitors only the lowest level of engagement, which is paying conscious, intentional attention to the content. In this paper, we propose a trajectory for visitor experience to a virtual heritage site that facilitates a higher level of engagement by letting visitors make their own interpretations. We developed a mobile virtual reality (VR) application that delivers this trajectory, which allows visitors to author mixed reality (MR) content that represents personal interpretations. The application provides three types of virtual assets to compose MR content: historical assets, emotional assets, and personal assets. We describe how visitors generally followed our trajectory and used virtual assets, engaging with virtual heritage and making interpretations. We related our findings to a discussion of how to support personal engagement and rich interpretation in a virtual heritage site.

3rd Digital Heritage International Congress (DigitalHERITAGE) held jointly with 2018 24th International Conference on Virtual Systems & Multimedia (VSMM 2018)

This paper proposes guidelines for the design of a Mixed Reality(MR) storytelling tool for cultural heritage sites that utilizes geotagged content and detailed location-specific narrative principles while prioritizing the intent of the author. Continuous efforts have been made to apply storytelling techniques in producing location-based digital heritage content over MR platforms. However, consideration for fine-tuned input parameters required at the authoring level has been consistently lacking. This has resulted in content that fails to identify, understand, and reflect the goals and needs of the author in maximizing the benefits of the narrative form for MR heritage experiences. To address this problem, we combine an analysis of existing location-based digital authoring tools with qualitative user studies conducted in a digital storytelling workshop. With the implications derived from our findings, we establish detailed design guidelines to provide a systematic narrative structure in the arrangement of geotagged content over various Points of Interest within the MR heritage space. Our study identifies two major authoring motivations for location-based MR storytelling: space-driven and story-driven. We thereby assert the need to bifurcate the design of the tool to support both these purposes and propose guidelines that differentiate the functions and task flow of each authoring mode.

3rd Digital Heritage International Congress (DigitalHERITAGE) held jointly with 2018 24th International Conference on Virtual Systems & Multimedia (VSMM 2018)

This study proposed a mobile virtual reality-based application for on-site risk management using Historical Building Information Modeling (HBIM) to alleviate the inefficiency of paper-based risk management of architectural heritage. In this research, we focused on the design of the metadata and the database structure that is based on the point of interest (PoI), anchor, and content metadata for advanced context-aware information retrieval from HBIM system in the mobile environment. To verify our method, we created a mobile VR-based application of the Gwandeokjeong building in Korea. After, we conducted a comparative experiment on performance time, task load and usability and proved the efficiency of our method for on-site risk management. In conclusion, our metadata and database structure in this study contribute to suggest a method to interoperate between HBIM system and mobile VR system and conduct advanced context-aware information retrieval.

3rd Digital Heritage International Congress (DigitalHERITAGE) held jointly with 2018 24th International Conference on Virtual Systems & Multimedia (VSMM 2018)

Despite various efforts to provide visually immersive experiences and various information about virtual cultural heritage sites, few studies have investigated how to effectively provide media related to cultural heritage sites in virtual reality or to measure the effectiveness of the virtual tourism experience in improving the affect and visit intention to the actual tourist destination. In this study, we examined the effects of a video-based virtual tour, applied via film-induced tourism, on post-VR attitude and behavioral visit intention to a cultural heritage site compared with those of a basic virtual tour. We also aimed to determine how to effectively provide broadcast content as a storytelling medium in a virtual reality tour, considering characteristics of a cultural heritage site. Through the experiment, we proved that video-based virtual experience is more effective in improving visitors' positive attitude and visit intention than the basic virtual tour, which was nevertheless effective as well. We also derived design implications for the utilization of location-based broadcast video clips.

IEEE International Symposium on Mixed and Augmented Reality (ISMAR) 2018Top-tier

We propose a novel method for model-based 3D tracking of hand articulations that is effective even for fastmoving hand postures in depth images. A large number of augmented reality (AR) and virtual reality (VR) studies have used model-based approaches for estimating hand postures and tracking movements. However, these approaches exhibit limitations if the hand moves rapidly or into the camera's field of view. To overcome these problems, researchers attempted a hybrid strategy that uses multiple initializations for 3D tracking of articulations. However, this strategy also exhibits limitations. For example, in genetic optimization, the hypotheses generated from the previous solution may search for a solution in an incorrect search space in a fast-moving hand gesture. This problem also occurs if the search space selected from the results of a trained model does not cover the true solution although the tracked hand moves slowly. Our proposed method estimates the hand pose based on model-based tracking guided by classification and search space adaptation. From the classification by a convolutional neural network (CNN), a data-driven prior is included in the objective function and additional hypotheses are generated in particle swarm optimization (PSO). In addition, the search spaces of the two sets of the hypotheses, generated by the data-driven prior and the previous solution, are adaptively updated using the distribution of each set of the hypotheses. We demonstrated the effectiveness of the proposed method by applying it to an American Sign Language (ASL) dataset consisting of fast-moving hand postures. The experimental results demonstrate that the proposed algorithm exhibits more accurate tracking results compared to other state-of-the-art tracking algorithms.

IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct) 2018

In this demonstration, we will show a prototype system with sensor fusion approach to robustly track 6 degrees of freedom of hand movement and support intuitive hand gesture interaction and 3D object manipulation for Mixed Reality head-mounted displays. Robust tracking of hand and finger with egocentric camera remains a challenging problem, especially with self-occlusion - for example, when user tries to grab a virtual object in midair by closing the palm. Our approach leverages the use of a common smart watch worn on the wrist to provide a more reliable palm and wrist orientation data, while fusing the data with camera to achieve robust hand motion and orientation for interaction.

IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct) 2018

We propose a method that is effective in tracking 3D hand poses occluded by a real object. Since existing model-based tracking methods rely only on observed images to estimate hand joints, tracking generally fails when the hand joints are largely invisible. This problem becomes more prevalent when the tracked hand is grabbing an object, as occlusion by the object makes it harder to find a proper correspondence between the hand model and observation. The proposed method utilizes the occluded part of the hand as additional information for model-based tracking. The occluded depth information is reconstructed according to the geometric of the object and model-based tracking is employed based on particle swarm optimization (PSO). We demonstrate that the reconstructed depth information improves the performance of tracking an object-grabbing hand.

Adjunct Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology

Augmented collaboration in a shared house design scenario has been studied widely with various approaches. However, those studies did not consider human perception. Our goal is to lower the user's perceptual load for augmented collaboration in shared space design scenarios. Applying attention theories, we implemented shared head gaze, shared selected object, and collaborative manipulation features in our system in two different versions with HoloLens. To investigate whether user perceptions of the two different versions differ, we conducted an experiment with 18 participants (9 pairs) and conducted a survey and semi-structured interviews. The results did not show significant differences between the two versions, but produced interesting insights. Based on the findings, we provide design guidelines for collaborative AR systems.

In International Conference on Human Interface and the Management of Information, Springer, Cham, 2018

In applying MR (AR/VR) technologies to cultural heritage sites, the design and implementation of mobile MR applications have mostly lacked a holistic and systematic approach in the viewpoint of retrieving existing content and providing personalized experiences to visitors on the fly. To address this issue, we designed and implemented the KCTM application in which various types of content from different databases are brought together and reorganized by a newly proposed metadata schema that assigns spatio-temporal information to them for the purpose of visualizing them in MR environments. According to the user evaluation, most of the participants expressed strong interest in touring the heritage sites augmented with various related content.

International Conference on Human Interface and the Management of Information (HIMI), pp. 188-201

As the popularization of augmented human, the influence of computing devices on human life is growing. We are facing lots of service in daily life, and it will be more in future. As a result, the border between real and virtual life is vanishing. To manage and maintain yourself in the flood of information, we need to know yourself in detail and quickly. For this reason, there were several research studies about the Quantified Self. However, the studies and applications of QS were held in separately by focusing each limited information, and it arises limitation on understanding in bird-eye view. To address these issues, we propose the unified concept of the Holistic Quantified Self (HQS) with its framework. HQS consists of the quantified indexes of user status which includes physical, emotional, social, and digital behavioral state. By collecting and quantifying the user’s multi-modal sensor data, HQS can be generated. Then, interpreting the pattern of HQS could be utilized to provide customized services to users. HQS framework describe an overall process to ensure the personalized service. Through these study, we expect to fulfill augmented human which enhances user abilities and contribute quality of social life through the development of the personal companion.

Proceedings of the 9th Augmented Human International Conference

In "Itchy Nose" we proposed a sensing technique for detecting finger movements on the nose for supporting subtle and discreet interaction. It uses the electrooculography sensors embedded in the frame of a pair of eyeglasses for data gathering and uses machine-learning technique to classify different gestures. Here we further propose an automated training and visualization tool for its classifier. This tool guides the user to make the gesture in proper timing and records the sensor data. It automatically picks the ground truth and trains a machine-learning classifier with it. With this tool, we can quickly create trained classifier that is personalized for the user and test various gestures.

IEEE International Conference on Consumer Electronics (ICCE) 2018

In this study, we designed an effective user interface (UIs) for lock screens using facial expression recognition technology and two interface methods to develop an application that induces positive emotional flow in children. There is not enough study considering practical use of lock screen with special characteristics for different types of user such as children. After that, in order to understand its effectiveness, the whole process on the methodology timeline consisted of two phases. As a result of the survey, children preferred a lot of tasks on the lock screen whereas adults who were the control group preferred a lower task load and were more likely to use the face recognition lock screen than the children. Based on the characteristics of the derived children, we expect to develop effective applications by integrating the results of the research and specifying the functions.

IEEE International Symposium on Mixed and Augmented Reality (ISMAR-Adjunct) 2017

We present `ARClassNote', an Augmented Reality application that enables users to save and share handwritten notes across multiple optical see-through Head Mounted Devices. In an augmented classroom environment, `ARClassNote' makes it easier to achieve bilateral communication between instructors and students, and share written class materials without occlusion. We showcase how each component of `ARClassNote' operates. We then propose suitable user interface for Augmented Reality head mounted devices. Evaluation of our applications' capabilities show that 'ARClassNote' is suitable for Augmented Reality Remote Education Solution.

IEEE International Symposium on Mixed and Augmented Reality (ISMAR-Adjunct) 2017

In this paper, we propose BoostHand, a freehand, distance-free object-manipulation system that supports simple trigger gestures using Leap Motion. In AR classrooms, it is necessary to allow both lecturers and students to utilize virtual teaching materials without any spatial restrictions, while handling virtual objects easily, regardless of distance. To provide efficient and accurate methods of handling AR classroom objects, our system requires only simple intuitive freehand gestures to control the users virtual hands in an enlarged, shared control space of users. We modified the GoGo interaction technique [5] by adding simple trigger gestures, and we evaluated performance against gaze-assisted selection (GaS) capabilities. Our proposed system enables both lecturers and students to utilize virtual teaching materials easily from their remote positions.

Proceedings of the 2017 ACM International Symposium on Wearable Computers

We propose a sensing technique for detecting finger movements on the nose, using EOG sensors embedded in the frame of a pair of eyeglasses. Eyeglasses wearers can use their fingers to exert different types of movement on the nose, such as flicking, pushing or rubbing. These subtle gestures can be used to control a wearable computer without calling attention to the user in public. We present two user studies where we test recognition accuracy for these movements.

International Symposium on Ubiquitous Virtual Reality (ISUVR) 2017

We presents a method to generate and track AR space using plane geometry information in AR environment. Our method calculates the major plane correspondences between frames and defines the relation. Using this relationship, we can create the virtual space that is unaffected by object position changes in images and can be used for AR tracking.

Proceedings of the 15th International Workshop on Content-Based Multimedia Indexing

This paper proposes a method to effectively utilize multimedia databases created and indexed by a metadata schema designed specifically for AR applications used at cultural heritage sites. We attempt to do so by incorporating storytelling principles that employ video data for the purpose of providing useful and meaningful guidance at Changdeokgung Palace, a UNESCO World Heritage Site that consists of multiple PoIs. We designed a themed narrative that embeds video data related to the PoIs, creating a guide route that connects each PoI in a fixed order. An extensive between-group user evaluation comparing a search-based AR experience and the proposed narrative-based one was conducted to test and prove the validity and effectiveness of our approach. Our results show that storytelling is indeed a significantly powerful tool to enhance the level of immersion users experience through video data in an AR environment at cultural heritage sites.

Distributed, Ambient and Pervasive Interactions

This paper presents an augmented reality (AR) authoring system that enables an ordinary user to easily build an AR environment by manipulating and placing 3D virtual objects. The system tracks users’ hand motions via an RGB-D camera which built-in an optical see-through (OST) head mounted display (HMD), and interactive features applied to virtual objects by the pre-defined hand gestures. In addition, the virtual objects and dynamic paths are placed by using a smartphone in a real-world space. To implement this system, three cored technologies are needed: (i) segmentation of regions of spaces and real-world objects (ii) hand tracking and gesture recognition for manipulating virtual objects, (iii) dynamic path placing of virtual objects using a smartphone wearing an OST HMD. We implement a prototype of the proposed system for testing its feasibility. To the end, we expect that our system enables simplify AR technology usage to ordinary users who are unfamiliar with professional AR authoring tools.

, International Conference on Distributed, Ambient and Pervasive Interactions (DAPI), held as part of the 19th International Conference on Human-Computer Interaction (HCI), pp.303-309, Oct, 2017

AR technology has been rapidly accepted in the cultural heritage domain, which requires wide context information for complete understanding as providing an enhanced experience to the user with related information of the physical world. However, currently, there are several limitations for seamless AR such as applications in outdoor environments and wearable platforms. To address this issue, we introduce our AR platform in this paper that supports the outdoor AR and wearable platform. Through the standardized metadata schema-based data retrieval, time-space correlated AR content containing the assorted context information can be provided to users. In addition, vision- and sensor-based spatial data composition technology supports stable 3D outdoor tracking. Finally, an immersive visualization provisioning module, which integrates the aforementioned component, provides a 360-degree panorama virtual reality for wearable platforms and the outdoor AR in mobile platforms. Using this platform, we expect a seamless context-aware AR service in the cultural heritage domain.

Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems

As mobile gaming has become increasingly popular in recent years, new forms of mobile game social media such as GameDuck that share mobile gameplay videos have emerged. In this work, we set out to understand the user motives of GameDuck by leveraging the Uses and Gratification Theory. We first explore the major motive themes from users' responses (n=138) and generate motivation survey items. We then identify the key motivators by conducting exploratory factor analysis of the survey results (n=354). Finally, we discuss how this new social media relates to existing systems such as Twitch.

Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems

TiTAN is a spatial user interface that enables freehand, midair text entry with a distant display while only requiring a low-cost depth sensor. Our system aims to leverage one's familiarity with the QWERTY layout. It allows users to input text, in midair, by mimicking the typing action they typically perform on a physical keyboard or touchscreen. Here, both hands and ten fingers are individually tracked, along with click action detection which enables a wide variety of interactions. We propose three midair text entry techniques and evaluate the TiTAN system with two different sensors.

IEEE International Conference on Consumer Electronics (ICCE) 2017

There have been a number of researches to provide new media experience through augmented reality technology. However, most of them focus on creating new virtual content rather than remediating existing content such as films and TV programs. In this paper, we design a metadata schema and use cases to provide location-based film experience based on the Spacetelling framework. To evaluate the metadata, we implement the application based on the relational database of the proposed metadata schema and provide it to visitors in the historical site. By analyzing users' requirements, we could find that the metadata properties provide new and valuable media experience in terms of affective, cognitive, and operative aspects. People felt more familiar with and interested in the place by watching movie clips in the related place. As a result, we suggest the design guideline derived from the experiment.

IEEE International Conference on Consumer Electronics (ICCE) 2017

Rediscovering Daereungwon is a location-based Augmented Reality(AR) mobile game application that aims to enrich the experience of navigating Daereungwon, a Korean cultural heritage site consisting of various royal tombs from the Silla dynasty. In this paper, a framework binding 3 main PoIs with distinctive features into a successive, cohesive, and immersive narrative is introduced. It appropriates and combines the Memorable Experience Design(MED) Framework and the Interest Curve proposed by Bulencea and Egger (2015) to guide tourists through Daereungwon, while game challenges and rewards in the form of augmented objects and interactions provide them with information on the PoIs and the site of Daereungwon as a whole.

12th International Conference on Signal-Image Technology & Internet-Based Systems (SITIS) 2016

In this paper, we propose an all-in-one mobile outdoor augmented reality (AR) framework for a cultural heritage site. The framework was designed to incorporate computer vision-based augmented reality technology and ontology-based data-authoring technology. Through this framework, we clearly explain how to create 3D visual data for camera pose estimation and how to connect AR content with a cultural heritage site. In addition, we suggest a multi-threading camera tracking and offer an estimation model for mobile AR application. Finally, we have confirmed the efficiency and reliability of our framework. Through this vision-based AR framework, seamless AR application for a cultural heritage site can be made.

12th International Conference on Signal-Image Technology & Internet-Based Systems (SITIS) 2016

In this study, we present a novel Augmented Reality (AR) application for providing contextual information of cultural heritage sites. Whereas most location-based AR systems are designed to show simple information on a Point of Interest (POI) in the real world, the suggested application offers information related to various cultural heritages including tangible and intangible heritages linked to the POI. is is accomplished by a cultural heritage ontology that aggregates heterogeneous data and reorganizes information in themes and relationships. The users can browse information such as the person who created the POI and events that took place at the location as separate entities. We implemented the application for Injeongjeon and the vicinity of Changdeokgung palace and conducted user studies to determine how people explore and consume contextual information at the heritage site. We provide directions that will be useful for designing information to support heritage site visiting, and discuss how the AR technology could improve the application.

Proceedings of the 18th ACM International Conference on Multimodal Interaction

Orbits combines a visual display and an eye motion sensor to allow a user to select between options by tracking a cursor with the eyes as the cursor travels in a circular path around each option. Using an off-the-shelf Jins MEME pair of eyeglasses, we present a pilot study that suggests that the eye movement required for Orbits can be sensed using three electrodes: one in the nose bridge and one in each nose pad. For forced choice binary selection, we achieve a 2.6 bits per second (bps) input rate at 250ms per input. We also inntroduce Head Orbits, where the user fixates the eyes on a target and moves the head in synchrony with the orbiting target. Measuring only the relative movement of the eyes in relation to the head, this method achieves a maximum rate of 2.0 bps at 500ms per input. Finally, we combine the two techniques together with a gyro to create an interface with a maximum input rate of 5.0 bps.

Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct

Solar System implements smooth pursuit eye movement interactions on commercial smart glasses using electrooculography. The system requires no calibration and little to no training. We present a prototype implementation, describe initial user tests and show several application scenarios for hands-free eye gaze interactions.

Proceedings of the 33rd Computer Graphics International

We propose an efficient method for model-based 3D tracking of hand articulations observed from an egocentric viewpoint that aims at supporting the manipulation of virtual objects. Previous model-based approaches optimize non-convex objective functions defined in the 26 Degrees of Freedom (DoFs) space of possible hand articulations. In our work, we decompose this space into six articulation subspaces (6 DoFs for the palm and 4 DoFs for each finger). We also label each finger with a Gaussian model that is propagated between successive image frames. As confirmed by a number of experiments, this divide-and-conquer approach tracks hand articulations more accurately than existing model-based approaches. At the same time, real time performance is achieved without the need of GPGPU processing. Additional experiments show that the proposed approach is preferable for supporting the accurate manipulation of virtual objects in VR/AR scenarios.

Distributed, Ambient and Pervasive Interactions

The Augmented Reality applications have received great attention in the recent years. However, there is still a lack of formal description of such systems currently. In this paper, we propose a new formal model for context-aware semantic Augmented Reality systems. The model can be divided into two parts: a set-theory function based method allows the formalization of an Augmented Reality system while an integrated time-space-motion logic provides the description of the behavior of the system. The suggested model enables the characterization of an Augmented Reality system with mathematical precision. In addition, logical inferences can be performed by means of the logic part of the formal description. The practical applicability of the proposed model is shown through use cases.

Distributed, Ambient and Pervasive Interactions

In this paper, we present a unified framework for remote collaboration using interactive augmented reality (AR) authoring and hand tracking methods. The proposed framework enables a local user to organize AR digital contents for making a shared working environment and collaborate multiple users in the distance. To develop the framework, we combine two core technologies: (i) interactive AR authoring method utilizing a smart input device for making a shared working space, (ii) hand-augmented object interaction method by tracking two hands in egocentric camera view. We implement a prototype of the proposed remote collaboration framework for testing its feasibility in an indoor environment. To the end, we expect that our framework enables collaboration as feeling a sense of co-presence with remote users in a user’s friendly AR working space.

Human Interface and the Management of Information: Applications and Services

In this paper, we introduce the K-Culture Time Machine Project, which develops a mobile AR platform for visualizing time-space connected cultural contents of Korea. Existing AR Applications in cultural heritage domains are currently not interoperable and cannot reuse content. To solve this problem, we developed the modified AR reference model as generic framework of a context-aware AR platform; and we developed the context-aware AR platform with several core technologies according to this model. For back-end, we established the Korean Cultural Heritage Data Model (KCHDM) to aggregate the heterogeneous cultural heritage databases in Korea. We also developed the semi-auto time-space correlation generation module for domain experts. Moreover, we developed the authoring tool to generate the time-space connected AR contents. For the front-end, we developed the vision- and sensor-based spatial data composition technology to perform the solid tracking in outdoor environment and context-aware AR framework. Through the UI/UX and 3D contents, whole technologies are packaged into the mobile AR platform. As a validation process for the application, a mobile AR application for the Korean world cultural heritage was developed. Based on this project, an interoperable AR platform that responds with heterogeneous database would be developed and smart tour guide of the cultural heritage site would be possible.

Distributed, Ambient and Pervasive Interactions

We present novel face landmark detection and tracking methods which are independent of user facial differences in a scenario of Spatial Augmented Reality (SAR) interaction. The proposed methods do not require a preliminary general face model to detect or track landmarks. Our contributions include: (i) fast face landmark detection, which is achieved based on our modified Latent Regression Forest (LRF) and (ii) model-independent facial landmark tracking by revising outliers based on a direction and displacement of neighboring landmarks. We also discuss (iii) feature enhancements based on RGB and depth images for supporting several interaction scenarios in SAR environments. We anticipate that the proposed methods promise several interesting scenarios, even under severe head orientation in SAR interaction without wearing any wearable devices.

Proceedings of the 2016 CHI Conference on Human Factors in Computing SystemsTop-tier

Virtual fitting rooms equipped with magic mirrors let people evaluate fashion items without actually putting them on. The mirrors superimpose virtual clothes on the user's reflection. We contribute the Mirror Mirror system, which not only supports mixing and matching of existing fashion items, but also lets users design new items in front of the mirror and export designs to fabric printers. While much of the related work deals with interactive cloth simulation on live user data, we focus on collaborative design activities and explore various ways of designing on the body with a mirror.

IEEE Second International Conference on Multimedia Big Data (BigMM) 2016

In recent years, augmented-reality (AR) has been attracting extensive attentions from both the research community and industry as a new form of media, mixing virtual content into the physical world. However, the scarcity of the AR content and the lack of user contexts are major impediments to providing and representing rich and dynamic multimedia content on AR applications. In this study, we propose an approach to search and filter big multimedia data, specifically geo-tagged mobile videos, for context-aware AR applications. The challenge is to automatically search for interesting video segments out of a huge amount of user-generated mobile videos, which is one of the biggest multimedia data, to be efficiently incorporated into AR applications. We model the significance of video segments as AR content adopting camera shooting patterns defined in filming, such as panning, zooming, tracking and arching. Then, several efficient algorithms are proposed to search for such patterns using fine granular geospatial properties of the videos such as camera locations and viewing directions over time. Experiments with real-world geo-tagged video dataset show that the proposed algorithms effectively search for a large collection of user-generated mobile videos to identify top K significant video segments.

IEEE Symposium on 3D User Interfaces (3DUI) 2016

We introduce a smartwatch assisted sensor fusion approach to robustly track 6-DOF hand movement in head mounted display (HMD) based augmented reality (AR) environment, which can be used for robust 3D object manipulation. Our method uses a wrist-worn smartwatch with HMD-mounted depth sensor to robustly track 3D position and orientation of user's hand. We introduce HMD-based augmented reality platform with smartwatch, and method to accurately calibrate orientation between smartwatch and HMD. We also implement natural 3D object manipulating system using 6-DOF hand tracker with hand grasping detection. Our proposed system is easy to use, and doesn't require any hand held devices.

Proceedings of the 7th Augmented Human International Conference 2016

This paper proposes a system which enables users to have enhanced interior deigning and authoring tool through augmented reality (AR). The proposed system, which we refer to as DreamHouse, focuses on providing natural user interaction and realistic AR experience by enabling following features: 1) allowing users to utilize bare-hand interaction with an attached egocentric RGB-D camera when trying to make detail adjustments to virtual objects' size, location and orientation, 2) rendering every virtual object in consideration of an environment and lighting conditions, creating photo-realistic scenes to help users to have immersive and realistic interior designing experience. As a result, DreamHouse allows users to freely and easily interact in physical space with virtual objects using bare hands and gives immersive and realistic AR authoring experience to users through photo-realistic rendering.