Publications & Patents

Proceedings of the 2026 CHI Conference on Human Factors in Computing SystemsTop-tier

Question-asking is one of the key indicators of cognitive engagement. However, understanding how the distinct psychological affordances of presentation media shape learners’ spoken inquiries with embodied Intelligent Virtual Agents (IVAs) remains limited. To systematically examine this process, we propose a 5W1H-based framework for analyzing learner questions. Using this framework, we conducted a user study comparing an Augmented Reality-based IVA (AR-IVA) deployed in the physical environment with a screen-based IVA (Video-IVA) during cardiopulmonary resuscitation (CPR) instruction. Results showed that the AR-IVA elicited higher spatial and social presence and promoted more frequent and longer questions focused on clarification and understanding. In contrast, the Video-IVA encouraged questions regarding procedural refinement. Presence acted as a selective filter, shaping the timing and …

IEEE Conference on Virtual Reality and 3D User Interfaces (VR) 2026Top-tier

We propose a novel object-mediated hand interaction system that enables real-time operation with everyday objects on wearable augmented reality (AR) devices. Despite recent advances, both commercial and academic hand interaction techniques remain constrained, typically requiring external hardware or depending exclusively on bare-hand gestures. Motivated by these constraints, we developed an offloading framework that integrates a high-fidelity transformer-based 3D hand reconstruction model with a dynamic gesture recognition network powered by gated recurrent units (GRU). This architecture ensures stable and accurate gesture recognition even during interaction with physical objects. To evaluate its quantitative performance, we collected a custom dataset based on a predefined gesture set, achieving 93.0% accuracy in 5-fold cross-validation. The complete system implemented on Microsoft HoloLens 2 operates at a real-time framerate, and we further analyze the latency of each step in our framework. Through this interaction paradigm, users can experience immersive and intuitive AR in everyday environments with minimal disruption to natural action behavior. Our projects are available at https://github.com/kaist-uvrlab/UnifiedHOInteraction.

IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW) 2026

We present a VR research demo for practical telepresence with photorealistic 3D Gaussian Splatting (3DGS) avatars under HMD-only sensing. The demo consists of two parts that provide complementary experiences in a single session: a face part, where HMD-provided blendshape signals drive an expressive 3DGS head avatar rendered stereoscopically on the client, and a full-body part, where a user-specific 3DGS full-body avatar is created from a single image and driven using HMD signals. During the demo, participants first capture a single image for full-body avatar creation, experience facial control using a pre-built third-person head avatar while the full-body avatar is generated, and then control their personalized full-body avatar in VR. By enabling both facial and full-body controllability and contrasting two rendering-responsibility designs, client-side direct 3D rendering and server-side stereoscopic image rendering/streaming-the demo offers practical insights for designing responsive, realistic VR telepresence systems under HMD-only sensing.

IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW) 2026

We present a real-time AR system that enables interaction with physical cuboid- and cylinder-shaped objects by representing them as compact superquadrics. Our approach uses Instant Surface Normal (ISN) maps-color images encoding per-pixel geometry-to detect and segment objects based on shape rather than appearance. Superquadric parameters are extracted sequentially, allowing efficient and updatable 3D modeling in real time. Experimental results show accurate detection, consistent segmentation, and reconstruction quality comparable to existing methods, while achieving over 5× faster inference. This system allows physical objects to be seamlessly integrated as interactive elements in dynamic AR environments.

IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW) 2026

We propose VR Zen Garden, a virtual reality (VR) application designed to support stress relief and meditation through interactive spatial soundscapes. The system enables users to create personalized auditory environments by combining two key interactions: Object Placement, which spatializes sound-emitting objects, and Sand Pattern Raking, which modulates sound properties through visual patterns. These inputs are processed by the Sound Spatializer and the Metaphoric Sound Mixer, allowing users to intuitively shape immersive 3D soundscapes in real time. VR Zen Garden offers an engaging and embodied approach to mindfulness, highlighting the potential of sound-driven design in immersive wellness applications.

IEEE Transactions on Visualization and Computer Graphics (Proc. IEEE VR'26)Top 10%

We present VRGaussianAvatar, an integrated system that enables real-time full-body 3D Gaussian Splatting (3DGS) avatars in virtual reality using only head-mounted display (HMD) tracking signals. The system adopts a parallel pipeline with a VR Frontend and a GA Backend. The VR Frontend uses inverse kinematics to estimate full-body pose and streams the resulting pose along with stereo camera parameters to the backend. The GA Backend stereoscopically renders a 3DGS avatar reconstructed from a single image. To improve stereo rendering efficiency, we introduce Binocular Batching, which jointly processes left and right eye views in a single batched pass to reduce redundant computation and support high-resolution VR displays. We evaluate VRGaussianAvatar with quantitative performance tests and a within-subject user study against image- and video-based mesh avatar baselines. Results show that VRGaussianAvatar sustains interactive VR performance and yields higher perceived appearance similarity, embodiment, and plausibility. Project page and source code are available at https://vrgaussianavatar.github.io.

HCI Korea 2026

SIGGRAPH Asia 2025 XR

We propose a collaborative scene mood authoring system in a virtual reality (VR) environment, where multiple users design and control the multimodal feedback (i.e., audiovisual and haptic) of a shared scene using voice commands. To support this, the user’s command is processed by large language models (LLMs) with the scene metadata and mood presets to identify intended target objects and mood, and the system modifies the objects to match the desired atmospheric expression. To cover a wide range of affective moods in scene mood authoring, we devised four distinct mood presets that represent different combinations of arousal and valence across the affective space. Each preset is expressed through a set of labeled visual, auditory, and haptic feedback elements, enabling coherent and multimodal expression of affective states within the scene.

IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct) 2025

We present a spatial augmentation system designed for Augmented Reality (AR) exhibition experiences, focusing on the immersive reinterpretation of two-dimensional artworks. Existing AR exhibition applications primarily augment content related to the artwork, but fall short of its architectural elements, which are crucial for integrating the artwork into the viewer’s surrounding space and enabling immersive spatial alignment. This study proposes a spatial augmentation approach that takes an artwork image as input, generates floor patterns and architectural 3D models through 2D and 3D model generation pipelines, and semantically aligns and augments them in the physical space. This spatial augmentation intends to enhance viewer engagement and perceived presence by maintaining spatial coherence between virtual and real space. By activating architectural surfaces as part of the experience, the system repositions AR exhibitions as spatial narratives rather than conventional overlay-driven displays.

IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct) 2025

We propose a lifestyle quantification method for personalized augmented reality (AR) intervention in everyday life. Maintaining a healthy lifestyle accompanies extensive workforce and costs, because accurate health evaluation and tailored management is a prerequisite. However, research on assessing multiple dimensions of health and timely administration of personalized information has been lacking. To address these issues, first we deduced participant’s physical activity, nutrition, interpersonal relations, and stress management level through their mobile devices. Each participant’s week-long data were filtered by clinical features evidenced in high-level medical literature. Next, based on their health status, we presented personalized AR interventions in their simulated everyday environment. Statistical analysis and user experiment results verified the clinical capability of our method and its effect on health-promoting decision behaviors of participants. Through this feasibility study, we hope to contribute on tailored lifestyle management in our daily lives, for better health of all.

IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct) 2025

We present a fast and efficient method for transferring facial textures onto SMPL-X–based full-body avatars. Unlike conventional affine-transform methods that are slow and prone to visual artifacts, our method utilizes a barycentric UV conversion technique. Our approach precomputes the entire UV mapping into a single transformation matrix, enabling texture transfer in a single operation. This results in a speedup of over 7000x compared to the baseline, while also significantly improving the final texture quality by eliminating boundary artifacts. Through quantitative and qualitative evaluations, we demonstrate that our method offers a practical solution for personalization in immersive XR applications. The code is available online.1

IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct) 2025

We present a multimodal framework that uses few-shot language models to generate action scene graphs from raw video for structuring spatial experiences in extended reality (XR). The system integrates unsupervised action segmentation, multimodal captioning with a vision-language model (VLM) guided by speech and motion cues, and scene graph generation from captions using a large language model (LLM), both leveraging few-shot prompting. By combining visual, verbal, and motion inputs, the framework produces semantically rich and accurate representations of spatial experiences without requiring annotated data or model fine-tuning. The resulting outputs will support a range of downstream XR applications, including adaptive playback, intent-driven retrieval, and flexible content reconstruction.

IEEE International Symposium on Mixed and Augmented Reality (ISMAR) 2025Top-tier

As immersive virtual environments become more prevalent, avatars serve as critical social interfaces. This study explores how combinations of visual appearance, vocal characteristics, and informed identity influence users' initial impressions and partner preferences in four distinct XR scenarios: physical, intellectual, social, and romantic. A within-subject experiment with 40 participants assessed perceived warmth, competence, uncanniness, and selection preferences across diverse avatar configurations. Results indicate that vocal cues had a particularly strong impact on social perception, often shaping feelings of approachability and clarity in communication. While some cue alignments enhanced perceived social comfort and engagement, inconsistencies across gender-related cues occasionally led to increased perceptions of uncanniness, especially in emotionally sensitive contexts. These findings highlight the importance of designing avatars that thoughtfully adapt to different interaction contexts, supporting inclusive and responsive user experiences in social XR platforms.

IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct) 2025

This paper presents a comprehensive framework for standardizing avatar face representation in Extended Reality (XR)-based communication. It defines a consistent representation model for facial features and expressions to ensure high visual fidelity and expressive accuracy across diverse XR devices. The proposed framework supports interoperability across different systems and applications by defining core data structures that adapt to the hardware constraints and capabilities of different setups. A key focus is the real-time synchronization of facial expressions, even when users’ faces are occluded by XR devices, to enhance social presence and interaction quality. To address variability in user devices and network conditions, the framework introduces an adaptive control mechanism for real-time avatar rendering. This mechanism dynamically adjusts the visual complexity and expressiveness of avatars based on device performance and network bandwidth, ensuring consistent and responsive communication experiences in heterogeneous environments. Ultimately, this standardization effort aims to advance immersive communication by enabling lifelike, responsive avatars that reflect users’ expressions, thereby accelerating the adoption of XR technologies in both professional and social domains.

IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct) 2025

This paper proposes a mutual space representation to support remote collaboration in Mixed and Augmented Reality (MAR) environments. To achieve high immersion and coexistence in MAR remote collaboration, a mutual space based on heterogeneous physical spaces that can be commonly utilized by collaborating participants is required. Although previous studies have proposed various spatial matching and modifying methods to overcome spatial heterogeneity, their different representations have made their integrated use in real-world scenarios difficult. To overcome these limitations, this paper proposes a mutual space representation standard that includes the representation of physical space, matching space, and collaboration context required for mutual space generation and management. We expect that the proposed standards will serve as a foundation for seamlessly applying solutions accumulated in academia to industrial systems, thereby providing users with a more immersive and coexistent MAR remote collaboration experience.

IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct) 2025

We propose a system that provides personalized interpretation of physical paintings using an AR cursor in a mixed reality (MR) environment. In galleries, pointing at specific regions of complex paintings is often restricted, making it hard to ask targeted questions. While prior works have proposed interactive systems to assist visitor understanding, they typically offered generalized explanations that were not grounded in real, physically exhibited artworks. To address this gap, our approach allows users to intuitively drag across the surface of a painting using a hand-guided AR cursor to specify a Region of Interest. The cropped image, along with the user’s spoken question, is then sent to a Vision Language Model (VLM), which generates a personalized and context-aware explanation based on the selected area. This approach enables users to gain a more precise and meaningful understanding of the painting, guided by their specific interests.

IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct) 2025

We propose Round-Trip2Gesture, an interaction system that uses smartwatch IMU sensors to enable inplace gesture input free from the sensing FOV of devices. Current smartglasses input is limited by social unsuitability of voice commands and by the restricted sensing field-of-view inherent to hand tracking. Our system introduces a delimiter gesture based on specific wrist motions to achieve reliable interaction in real-world scenarios. Our system enables intuitive gesture interaction free from occulsion and physical constraints. We evaluated a set of 5 gestures with 8 participants. Four gestures showed over 90% accuracy in leave-one-subject-out cross-validation. We introduce the potential of the system through notification dismissal and video playback control applications, demonstrating seamless out-of-FOV interaction in Mixed Reality environments.

IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct) 2025

We propose a sequential context engineering framework for zero-shot recognition of procedural tasks in egocentric AR. While large vision-language models excel at zero-shot image classification, they lack temporal reasoning required for procedural understanding, and fully fine-tuning them degrades generalization capabilities. The proposed method preserves frozen CLIP encoders while introducing lightweight modules: an intra-action attention module for temporal aggregation and an inter-action context module that leverages previous predictions to generate context-aware prompts. Evaluated on EGTEA Gaze+ dataset, our approach shows promising results with notable improvements over baselines while maintaining parameter efficiency. This work presents a step toward enabling temporal reasoning for VLMs in resource-constrained AR systems without sacrificing zero-shot generalization.

IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct) 2025

We propose the Spatio-Temporal Mixed and Augmented Reality Experience Description (MAR-ED), a novel framework to standardize the representation of past events for interactive and adaptive playback in a user’s present physical space. While current spatial media technologies have primarily focused on capturing or replaying content as static assets, often disconnected from the viewer’s environment or offering limited interactivity, the means to describe an experience’s underlying semantic and interactive structure remains underexplored. We propose a descriptive framework called MARED based on three core primitives: 1) Event Primitives for semantic scene graph representation, 2) Keyframe Primitives for efficient and meaningful data access, and 3) Playback Primitives for user-driven adaptive interactive playback of recorded MAR experience. The proposed flowchart of the three-stage process of the proposed MAR-ED framework transforms a recorded experience into a unique adaptive MAR experience during playback, where its spatio-temporal structure dynamically conforms to a new environment and its narrative can be altered by live user input. Drawing on this framework, personal digital memories and recorded events can evolve beyond passive 2D/3D videos into immersive, spatially-integrated group experiences, opening new paradigms for training, cultural heritage, and interactive storytelling without requiring complex, per-user adaptive rendering.

Adjunct Proceedings of the 38th Annual ACM Symposium on User Interface Software and Technology

Figure 1: RealityCrafter creates an editable 3D scene driven by user speech from a single real-world image.Our system (a) takes a single image as input; (b) reconstructs it into a 3D scene; and (c) empowers users to author that space via voice commands and pointer interactions.

Proceedings of the 2025 CHI Conference on Human Factors in Computing SystemsTop-tier

This research investigates the use of hybrid user interfaces to enhance text readability in augmented reality (AR) by combining optical see-through head-mounted displays with smartphones. While this integration can improve information legibility, it may also introduce display switching side effects. The extent to which these side effects hinder user experience and when the benefits outweigh drawbacks remain unclear. To address this gap, we conducted an empirical study (N=24) to evaluate how hybrid user interfaces affect AR reading tasks across different content distances, which induce varying levels of display switching. Our findings show that hybrid user interfaces offer significant readability benefits compared to using the HMD only, reducing mental and physical demands when reading text linked to content at closer distances. However, as the distance between displays increases, the compensatory behaviors users adopt to manage increased switching costs negate these benefits, making hybrid user interfaces less effective. Based on these findings, we suggest (1) using smartphones as supplementary displays for text in reading-intensive tasks, (2) implementing adaptive display positioning to minimize switching overhead in such scenarios, and (3) adjusting the smartphone’s role based on content distance for less intensive reading tasks. These insights provide guidance for optimizing smartphone integration in hybrid interfaces and enhancing AR systems for reading applications.

IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW) 2025

Asynchronous communication is essential in diverse contexts such as remote meetings, education, and collaboration. However, traditional replay-based systems lack interactivity and often fail to preserve critical contextual cues. This paper introduces Avagent, a novel framework for asynchronous com munication in Virtual/Augmented Reality (VR/AR) environments that leverages agentized avatars to bridge temporal and contextual gaps between participants. Unlike typical agent-mediated systems, the framework utilizes users’ data to create Avagents that reflect their actual intent, emotional states, and behaviors. By accurately capturing and replicating both verbal and nonverbal behaviors, Avagents enable more interactive and context-rich communication. This approach facilitates dynamic interactions, enhances social presence, and ensures continuity across timelines. Avagents empower new users to engage interactively with prior discussions, fostering deeper understanding and seamless collaboration with previous users. Envisioning the benefits of Avagents as an engaging and context-rich solution for asynchronous communication, this paper outlines interaction scenarios, a work-in-progress prototype, and associated challenges.

IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW) 2025

We present a new approach using Virtual Reality to capture human-scene interaction data within 3D environments. Traditional methods required costly physical setups with real furniture, making it difficult to collect both interaction and 3D scene data comprehensively. Our solution implements a user-centric scene graph with a hierarchical structure that captures both user interactions and 3D scene elements. The resulting dataset contains real human interaction data, dynamic scene information, synthetic sensor data, and scene graph annotations. This work provides a foundation for developing Mixed Reality experiences using 3D scene graphs with human interaction data.

한국소프트웨어종합학술대회 (KSC 2025)

IEEE Transactions on Visualization and Computer GraphicsTop 10%

Immersive technologies such as virtual and augmented reality (VR/AR) allow remote users to meet and interact in a shared virtual space using embodied virtual avatars, creating a sense of co-presence. However, asynchronous communication-essential in many real-world contexts-remains underexplored in these environments. Traditional playback-based systems lack interactivity and often fail to preserve critical contextual cues necessary for effective asynchronous communication. In this paper, we introduce AVAGENTs, AI-powered virtual avatars that replicate users' verbal and nonverbal cues from recordings of past meetings. Avagents can interpret meeting context and generate appropriate responses to questions posed by asynchronous viewers. Through a user study (N = 30), we evaluated Avagents against a traditional playback method and a voice-based AI assistant across two asynchronous meeting scenarios: analytic reasoning and affective resonance. Results showed that Avagents enhance the asynchronous communication experience by increasing social presence, sense of belonging, emotional intimacy, and other user perceptions. We discuss the findings and their implications for designing effective AI-driven asynchronous communication tools in VR/AR environments.

IEEE Transactions on Visualization and Computer Graphics (Proc. IEEE VR'25)Top 10%

As avatars have evolved from simple digital representations into extensions of our identities, they offer unprecedented opportunities for self-expression and customization beyond the physical world limitations. While virtual platforms foster new forms of identity exploration, social norms still play a crucial role in defining what is considered appropriate in these environments. In this study, we surveyed 150 participants to investigate social norms surrounding avatar modifications, examining how perspectives, contexts, and personality traits influence attitudes toward appropriateness. Our findings reveal that avatar modifications are generally viewed as more appropriate when considered from a partner's perspective, especially for changeable attributes. However, these modifications are perceived as less acceptable in professional settings such as workplaces. Additionally, individuals with high self-monitoring tendencies tend to be more resistant to changes, while those scoring higher on Machiavellianism are more accepting of changes, particularly regarding unchangeable attributes and emotional expressions. These findings provide valuable insights for platform developers and designers, highlighting the importance of implementing context-aware customization options that balance core identity elements with personality-driven preferences, thereby enhancing user experiences while respecting social norms.

IEEE Computer Graphics and Applications

We introduce the concept of a meta-object, a next-generation virtual object that inherits the form, properties, and functions of its real-world counterpart, enabling seamless synchronization, interaction, and sharing between the physical and virtual worlds. While plenty of today's virtual objects provide some sensory feedback and dynamic behavior, meta-objects fully integrate interactive and multisensory features within a structured data framework to enable real-time immersive experiences in a postmetaverse intelligent simulation platform. Three key components underpin the utilization of meta-objects in the postmetaverse: property-embedded modeling for physical and action realism, adaptive multisensory feedback tailored to user interactions, and a scene graph-based intelligence simulation platform for scalable and efficient ecosystem integration. By leveraging meta-objects through wearable augmented reality/virtual reality devices, the postmetaverse facilitates seamless interactions that transcend spatial and temporal barriers, paving the way for a transformative reality-virtuality convergence.

GMCV Workshop at CVPR'25 (CVPRW)

Generating high-quality 3D models from 2D sketches is a challenging task due to the inherent ambiguity and sparsity of sketch data. In this paper, we present S3D, a novel framework that converts simple hand-drawn sketches into detailed 3D models. Our method utilizes a U-Net-based encoder-decoder architecture to convert sketches into face segmentation masks, which are then used to generate a 3D representation that can be rendered from novel views. To ensure robust consistency between the sketch domain and the 3D output, we introduce a novel style-alignment loss that aligns the U-Net bottleneck features with the initial encoder outputs of the 3D generation module, significantly enhancing reconstruction fidelity. To further enhance the network's robustness, we apply augmentation techniques to the sketch dataset. This streamlined framework demonstrates the effectiveness of S3D in generating high-quality 3D models from sketch inputs. The source code for this project is publicly available at https://github.com/hailsong/S3D.

IEEE International Symposium on Mixed and Augmented Reality (ISMAR) 2024Top-tier

We propose a novel crowd data-driven optimization approach for artwork placement in virtual exhibitions. With the emerging concept of Metaverse, a multitude of users can engage with content contemporaneously in virtual exhibitions. Yet, few studies have suggested a method to resolve crowd density concentration in multiuser Mixed Reality (MR) and Virtual Reality (VR) environments. In this study, our approach leverages crowd data engaged with artworks to predict optimal placement of artworks to distribute crowd density in virtual exhibitions, prior to exhibition planning. To investigate the requirement and validity of our approach, we conducted focus group interviews and an artwork relocation experiment as preliminary studies. In the generation of solution scenes for optimal placement, our optimizer adaptively scrutinizes placeable areas with considerations of crowd density distribution, scene rationality, and artwork similarity. Through a performance comparison analysis between optimization results, we confirmed that the optimizer successfully fulfilled the intended objectives with respect to the design considerations, resolving practical scenarios in exhibition planning.

IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct) 2024

We propose an algorithm that extracts the most suitable affordances, interaction targets, and corresponding coordinates adaptively from 3D models of various artifacts based on their functional context for efficient authoring of XR content with artifacts. Traditionally, authoring AR scenes to convey artifact context required one-to-one manual work. Our approach leverages a Large Language Model (LLM) to extract interaction types, positions, and subjects based on the artifact's name and usage context. This enables templated XR experience creation, replacing repetitive manual labor. Consequently, our system streamlines the XR authoring process, making it more efficient and scalable.

IEEE International Symposium on Mixed and Augmented Reality (ISMAR) 2024Top-tier

Understanding gender differences in facial and spatial recognition is crucial for enhancing avatar-mediated communication. However, there remains a gap in understanding how participant gender influences perceptions of avatar facial expressions and spatial dynamics in Mixed Reality communication. Therefore, our study investigates how avatar non-verbal cues interact with gender differences to affect user experience and understanding in MR environments. To examine these complex relationships, we conducted a user study comparing the effects of various avatar facial expressions (Full, Mouth-Only, and Emotion-based) and interpersonal distances (Closer vs. Farther) on facial animation realism and social presence, with a focus on gender-balanced participant groups. Our findings revealed that female participants were particularly sensitive to the avatar’s proximity and facial expressions, reporting significantly higher perceptions of facial animation realism, copresence, message understanding, and affective understanding at farther distances compared to male participants. They also perceived higher copresence and message understanding when exposed to emotion-based facial expressions, as opposed to a mouth-only condition-a distinction not observed among male participants. Based on our findings, we advocate for avatar design strategies that accommodate gender differences in perception and preference, potentially through customizable levels of expressiveness to cater to diverse user needs and contexts.

IEEE International Symposium on Mixed and Augmented Reality (ISMAR) 2024Top-tier

To enable remote Virtual Reality (VR) and Augmented Reality (AR) clients to collaborate as if they were in the same space during Mixed Reality (MR) telepresence, it is essential to overcome spatial heterogeneity and generate a unified shared collaborative environment by integrating remote spaces into a target host space. Especially when multiple remote users connect, a large shared space is necessary for people to maintain their personal space while collaborating, but the existing simple intersection method leads to the creation of narrow shared spaces as the number of remote spaces increases. To robustly align to the host space even as the number of remote spaces increases, we propose a spatial affordance-aware interactable subspace allocation algorithm. The key concept of our approach is to consider the perceivable and interactable areas separately, where every user views the same mutual space, but each remote user has a different interactable subspace, considering their location and spatial affordance. We conducted an evaluation with 900 space combinations, varying the number of remote spaces as two, four, and six, and results show our method outperformed in securing wide interactable mutual space and instantiating users compared to the other spatial matching methods. Our work enables multiple clients from diverse remote locations to access the AR host’s space, allowing them to interact directly with the table, wall, or floor by aligning their physical subspaces within a connected mutual space.

IEEE International Symposium on Mixed and Augmented Reality (ISMAR) 2024Top-tier

In avatar-mediated remote communication, avatars’ facial expressions can be dynamically adjusted according to each user’s computational and device constraints, highlighting the importance of varied expressions and their impact on user perception. However, there is a lack of research on how variations in avatar facial expressions, especially when simplified, influence user perception, particularly in terms of social presence. To address this, we examine the impact of various facial expression combinations on social presence in avatar-mediated communication scenarios, ranging from informative speeches to emotional conversations. Our approach involves prioritizing avatar facial blendshape combinations using two main approaches: (1) commonly activated expressions that reflect the active facial movements observed during casual conversations, and (2) emotion-based expressions derived from Facial Action Coding System (FACS). These combinations were compared against minimal baseline and full blendshape conditions through a comprehensive study involving 32 participants. Our findings reveal that emotion-based condition achieves comparable levels of social presence and communication quality to the full condition, in both informative speeches and emotional conversations. This highlights the effectiveness of prioritizing emotion-based expressions and adopting a streamlined approach to avatar facial control. By focusing on emotional expressions while optimizing resources, this approach shows potential for enhancing the avatar-mediated communication experience, accommodating the diverse users’ contexts.

IEEE International Symposium on Mixed and Augmented Reality (ISMAR) 2024Top-tier

We introduce “Whirling Interface,” a selection method for XR displays using bare-hand motion matching gestures as an input technique. We extend the motion matching input method, by introducing different input states to provide visual feedback and guidance to the users. Using the wrist joint as the primary input modality, our technique reduces user fatigue and improves performance while selecting small and distant targets. In a study with 16 participants, we compared the whirling interface with a standard ray casting method using hand gestures. The results demonstrate that the Whirling Interface consistently achieves high success rates, especially for distant targets, averaging 95.58% with a completion time of 5.58 seconds. Notably, it requires a smaller camera sensing field of view of only 21.45° horizontally and 24.7° vertically. Participants reported lower workloads on distant conditions and expressed a higher preference for the Whirling Interface in general. These findings suggest that the Whirling Interface could be a useful alternative input method for XR displays with a small camera sensing FOV or when interacting with small targets.

Adjunct Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology

Diminished reality (DR) removes virtual and real objects from the user’s view, decluttering and optimizing what the user sees through augmented/mixed reality. However, removing real objects create a potential safety concern because users may unintentionally bump into the diminished object. To address this issue, we apply characteristics of peripheral vision to DR and diminish objects when they are in the peripheral region and show the object when the user focuses on the object. We created 3 different object opacity control functions that use gaze information and evaluated them.

European Conference on Computer Vision (ECCV) 2024

Existing datasets for 3D hand-object interaction are limited either in the data cardinality, data variations in interaction scenarios, or the quality of annotations. In this work, we present a comprehensive new training dataset for hand-object interaction called HOGraspNet. It is the only real dataset that captures full grasp taxonomies, providing grasp annotation and wide intraclass variations. Using grasp taxonomies as atomic actions, their space and time combinatorial can represent complex hand activities around objects. We select 22 rigid objects from the YCB dataset and 8 other compound objects using shape and size taxonomies, ensuring coverage of all hand grasp configurations. The dataset includes diverse hand shapes from 99 participants aged 10 to 74, continuous video frames, and a 1.5M RGB-Depth of sparse frames with annotations. It offers labels for 3D hand and object meshes, 3D keypoints, contact maps, and grasp labels. Accurate hand and object 3D meshes are obtained by fitting the hand parametric model (MANO) and the hand implicit function (HALO) to multi-view RGBD frames, with the MoCap system only for objects. Note that HALO fitting does not require any parameter tuning, enabling scalability to the dataset’s size with comparable accuracy to MANO. We evaluate HOGraspNet on relevant tasks: grasp classification and 3D hand pose estimation. The result shows performance variations based on grasp type and object class, indicating the potential importance of the interaction space captured by our dataset. The provided data aims at learning universal shape priors or foundation models for 3D hand-object interaction. Our dataset and code are available at https://hograspnet2024.github.io/.

Springer Virtual Reality

We propose a robust 3D hand tracking system in various hand action environments, including hand-object interaction, which utilizes a single color image and a previous pose prediction as input. We observe that existing methods deterministically exploit temporal information in motion space, failing to address realistic diverse hand motions. Also, prior methods paid less attention to efficiency as well as robust performance, i.e., the balance issues between time and accuracy. The Temporally Enhanced Graph Convolutional Network (TE-GCN) utilizes a 2-stage framework to encode temporal information adaptively. The system establishes balance by adopting an adaptive GCN, which effectively learns the spatial dependency between hand mesh vertices. Furthermore, the system leverages the previous prediction by estimating the relevance across image features through the attention mechanism. The proposed method achieves state-of-the-art balanced performance on challenging benchmarks and demonstrates robust results on various hand motions in real scenes. Moreover, the hand tracking system is integrated into a recent HMD with an off-loading framework, achieving a real-time framerate while maintaining high performance. Our study improves the usability of a high-performance hand-tracking method, which can be generalized to other algorithms and contributes to the usage of HMD in everyday life. Our code with the HMD project will be available at https://github.com/UVR-WJCHO/TEGCN_on_Hololens2 .

Proceedings of the CHI Conference on Human Factors in Computing SystemsTop-tier

We investigate how path context, encompassing both comfort and attractiveness, shapes user experiences in outdoor locative storytelling using Augmented Reality (AR). Addressing a research gap that predominantly concentrates on indoor settings or narrative backdrops, our user-focused research delves into the interplay between perceived path context and locative AR storytelling on routes with diverse walkability levels. We examine the correlation and causation between narrative engagement, spatial presence, perceived workload, and perceived path context. Our findings show that on paths with reasonable path walkability, attractive elements positively influence the narrative experience. However, even in environments with assured narrative walkability, inappropriate safety elements can divert user attention to mobility, hindering the integration of real-world features into the narrative. These results carry significant implications for path creation in outdoor locative AR storytelling, underscoring the importance of ensuring comfort and maintaining a balance between comfort and attractiveness to enrich the outdoor AR storytelling experience.

Proceedings of the CHI Conference on Human Factors in Computing SystemsTop-tier

Augmented Reality (AR) is regarded as an innovative storytelling medium that presents novel experiences by layering a virtual narrative space over a real 3D space. However, understanding of how the virtual narrative space and the real space are connected with one another in the design of augmented narrative spaces has been limited. For this, we conducted a systematic literature review of 64 articles featuring AR storytelling applications and systems in HCI, AR, and MR research. We investigated how virtual narrative spaces have been paired, functionalized, placed, and registered in relation to the real spaces they target. Based on these connections, we identified eight dominant types of augmented narrative spaces that are primarily categorized by whether they virtually narrativize reality or realize the virtual narrative. We discuss our findings to propose design recommendations on how virtual-real connections can be incorporated into a more structured approach to AR storytelling.

Proceedings of the Augmented Humans International Conference 2024

We propose GestureMark, a novel input technique for target selection on XR glasses using smartwatch touch gestures as input. As XR glasses get smaller and lighter, their usage increases rapidly, leading to a higher demand for efficient shortcuts for everyday life. We explored the uses of gesture input on smartwatch touchscreen, including simple swipe, swipe combinations, and bezel-to-bezel (B2B) gesture as an input modality. Through an experiment with 16 participants, we found that while swipe gestures were efficient for four-choice selections, B2B was superior for 16-choice inputs. Feedback mechanisms did not enhance performance but reduced perceived workload. Our findings highlight the potential of integrating smartwatches as secondary input devices for XR glasses.

Proceedings of the Augmented Humans International Conference 2024

Assessing a patient’s social and environmental factors is crucial in medical treatment due to their impact on overall quality of life and treatment outcomes. However, current industrial solutions and research on augmented reality and virtual reality telehealth have primarily focused on training or physician collaboration, thereby overlooking the importance of assessing patients’ non-medical factors in the consulting environment. To address this issue, we propose a holistic patient assessment system in an extended reality (XR) teleconsultation environment, based on clinical evidence from the literature. By utilizing the digital twin (DT) of the patient’s environment and a physical examination assistance tool, healthcare professionals can effectively assess non-medical factors and enhance diagnosis accuracy. Through evaluations by current healthcare professionals, our system has been medically validated for its adequate integration of crucial factors in medicine and potential for real-world deployment. We believe that our system could envision a new line of research in future XR telehealth, particularly in the context of patient-centered and inclusive care.

IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW) 2024

This study proposes an experience graph, a Mixed Reality (MR) experience replaying method for recording and playing spatially adaptive interaction in a mixed reality environment. Many MR immersive interactions are experienced in a space different from the original recorded space, so it is important to adaptively experience the interaction in the local space. As a solution to the collision and mismatch during the interactions in local space, there is research on scene generation or adaptive object placement, but the change of interactions dependent on space and the resulting experience transformation were not considered. This study structures spatial information and dependent interactions into an experience graph and proposes a spatial adaptive replay method that minimizes interaction deformation. Recorded interactions are evaluated quantitatively through loss rate and transformation rate, played interactions are evaluated qualitatively through user experiments, and finally, indicators important to the user experience are reflected in the experience graph. Utilizing the results of this study will contribute to experiencing immersive spatial MR where users actively record and share spatial experiences beyond the limitations of time and space.

IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW) 2024

In avatar-mediated remote communication, reflecting one's facial expressions in avatars remains challenging but essential in facilitating effective communication and fostering social presence. From our prior studies, we underscored the significance of emotion-based facial expressions in avatar-mediated remote communications within eXtended Reality (XR). Our findings indicated that during emotional conversations as well as informative speeches, activating the emotion-based blendshapes resulted in a level of social presence and communication quality comparable to activating all facial blend-shapes. As we move forward, our research aims to investigate avatar facial expression considering XR contexts, with an emphasis on the challenges stemming from XR device constraints and the dynamic contexts of remote collaboration. We aspire to devise adaptive guidelines for avatar facial expression, leveraging emotional cues using XR devices. Additionally, we plan to systematically analyze the implications of remote collaboration's varying scenarios, formulating guidelines for adaptive avatar representation that account for different collaborative environments and scenarios. To enhance social presence and communication experiences among users, we are preparing for a series of user studies, aiming to validate the effectiveness of our approaches and prototype systems.

IEEE Conference on Virtual Reality and 3D User Interfaces (VR) 2024Top-tier

We propose Object Cluster Registration (OCR) using Geometric Spatial Affordance Graph (GSAG) to support user interaction with multiple objects in a shared space generated from two dissimilar rooms. Previous research on generating a shared virtual space from dissimilar real spaces has only reflected the information of individual objects and aimed at maximizing the area of the shared space. This led to limited interactions relying on the singular affordances of objects, neglecting to consider the usability and effectiveness of the generated shared spaces. The proposed OCR with GSAG, which considers the relationship between objects based on facing formation, extracts optimal object cluster pairs to align dissimilar rooms in generating shared virtual spaces. In an evaluation study involving 100 multi-cluster space pairs, applying OCR using GSAG showed greater effectiveness in preserving object correlations compared to cases where OCR was not used. Furthermore, the size of the shared space did not significantly differ between the two methods. This suggests that factoring in the relationship between objects does not compromise the objective of maximizing the shared virtual space. The proposed method is expected to serve as a foundation for generating shared virtual spaces that are more user-oriented and efficient by facilitating a wider range of collaborative activities for remote users in dissimilar real spaces with varied configurations.

IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW) 2024

Realistic avatars are fundamental for immersive experiences in Augmented Reality (AR) and Virtual Reality (VR) environments. In this work, we introduce a novel approach for avatar generation, combining 3D Gaussian Splatting with the parametric body model, SMPL. This methodology overcomes the inefficiencies of traditional image/video-based avatar creation, which is often slow and requires high computing resources. The integration of 3D Gaussian Splatting for representing human avatar offers realistic and real-time rendering for AR/VR applications. We also conducted preliminary tests to verify the quality of avatar representation using 3D Gaussian Splatting. These tests, displayed alongside outcomes from existing methods, demonstrate the potential of this research to significantly contribute to the creation of realistic avatars in the future. Additionally, several key discussions are presented, essential for developing and evaluating the system and providing valuable insights for future research.