Publications & Patents

Proceedings of the 2026 CHI Conference on Human Factors in Computing SystemsTop-tier

Question-asking is one of the key indicators of cognitive engagement. However, understanding how the distinct psychological affordances of presentation media shape learners’ spoken inquiries with embodied Intelligent Virtual Agents (IVAs) remains limited. To systematically examine this process, we propose a 5W1H-based framework for analyzing learner questions. Using this framework, we conducted a user study comparing an Augmented Reality-based IVA (AR-IVA) deployed in the physical environment with a screen-based IVA (Video-IVA) during cardiopulmonary resuscitation (CPR) instruction. Results showed that the AR-IVA elicited higher spatial and social presence and promoted more frequent and longer questions focused on clarification and understanding. In contrast, the Video-IVA encouraged questions regarding procedural refinement. Presence acted as a selective filter, shaping the timing and …

IEEE Conference on Virtual Reality and 3D User Interfaces (VR) 2026Top-tier

We propose a novel object-mediated hand interaction system that enables real-time operation with everyday objects on wearable augmented reality (AR) devices. Despite recent advances, both commercial and academic hand interaction techniques remain constrained, typically requiring external hardware or depending exclusively on bare-hand gestures. Motivated by these constraints, we developed an offloading framework that integrates a high-fidelity transformer-based 3D hand reconstruction model with a dynamic gesture recognition network powered by gated recurrent units (GRU). This architecture ensures stable and accurate gesture recognition even during interaction with physical objects. To evaluate its quantitative performance, we collected a custom dataset based on a predefined gesture set, achieving 93.0% accuracy in 5-fold cross-validation. The complete system implemented on Microsoft HoloLens 2 operates at a real-time framerate, and we further analyze the latency of each step in our framework. Through this interaction paradigm, users can experience immersive and intuitive AR in everyday environments with minimal disruption to natural action behavior. Our projects are available at https://github.com/kaist-uvrlab/UnifiedHOInteraction.

IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW) 2026

We present a VR research demo for practical telepresence with photorealistic 3D Gaussian Splatting (3DGS) avatars under HMD-only sensing. The demo consists of two parts that provide complementary experiences in a single session: a face part, where HMD-provided blendshape signals drive an expressive 3DGS head avatar rendered stereoscopically on the client, and a full-body part, where a user-specific 3DGS full-body avatar is created from a single image and driven using HMD signals. During the demo, participants first capture a single image for full-body avatar creation, experience facial control using a pre-built third-person head avatar while the full-body avatar is generated, and then control their personalized full-body avatar in VR. By enabling both facial and full-body controllability and contrasting two rendering-responsibility designs, client-side direct 3D rendering and server-side stereoscopic image rendering/streaming-the demo offers practical insights for designing responsive, realistic VR telepresence systems under HMD-only sensing.

IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW) 2026

We present a real-time AR system that enables interaction with physical cuboid- and cylinder-shaped objects by representing them as compact superquadrics. Our approach uses Instant Surface Normal (ISN) maps-color images encoding per-pixel geometry-to detect and segment objects based on shape rather than appearance. Superquadric parameters are extracted sequentially, allowing efficient and updatable 3D modeling in real time. Experimental results show accurate detection, consistent segmentation, and reconstruction quality comparable to existing methods, while achieving over 5× faster inference. This system allows physical objects to be seamlessly integrated as interactive elements in dynamic AR environments.

IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW) 2026

We propose VR Zen Garden, a virtual reality (VR) application designed to support stress relief and meditation through interactive spatial soundscapes. The system enables users to create personalized auditory environments by combining two key interactions: Object Placement, which spatializes sound-emitting objects, and Sand Pattern Raking, which modulates sound properties through visual patterns. These inputs are processed by the Sound Spatializer and the Metaphoric Sound Mixer, allowing users to intuitively shape immersive 3D soundscapes in real time. VR Zen Garden offers an engaging and embodied approach to mindfulness, highlighting the potential of sound-driven design in immersive wellness applications.

IEEE Transactions on Visualization and Computer Graphics (Proc. IEEE VR'26)Top 10%

We present VRGaussianAvatar, an integrated system that enables real-time full-body 3D Gaussian Splatting (3DGS) avatars in virtual reality using only head-mounted display (HMD) tracking signals. The system adopts a parallel pipeline with a VR Frontend and a GA Backend. The VR Frontend uses inverse kinematics to estimate full-body pose and streams the resulting pose along with stereo camera parameters to the backend. The GA Backend stereoscopically renders a 3DGS avatar reconstructed from a single image. To improve stereo rendering efficiency, we introduce Binocular Batching, which jointly processes left and right eye views in a single batched pass to reduce redundant computation and support high-resolution VR displays. We evaluate VRGaussianAvatar with quantitative performance tests and a within-subject user study against image- and video-based mesh avatar baselines. Results show that VRGaussianAvatar sustains interactive VR performance and yields higher perceived appearance similarity, embodiment, and plausibility. Project page and source code are available at https://vrgaussianavatar.github.io.

HCI Korea 2026

SIGGRAPH Asia 2025 XR

We propose a collaborative scene mood authoring system in a virtual reality (VR) environment, where multiple users design and control the multimodal feedback (i.e., audiovisual and haptic) of a shared scene using voice commands. To support this, the user’s command is processed by large language models (LLMs) with the scene metadata and mood presets to identify intended target objects and mood, and the system modifies the objects to match the desired atmospheric expression. To cover a wide range of affective moods in scene mood authoring, we devised four distinct mood presets that represent different combinations of arousal and valence across the affective space. Each preset is expressed through a set of labeled visual, auditory, and haptic feedback elements, enabling coherent and multimodal expression of affective states within the scene.

IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct) 2025

We present a spatial augmentation system designed for Augmented Reality (AR) exhibition experiences, focusing on the immersive reinterpretation of two-dimensional artworks. Existing AR exhibition applications primarily augment content related to the artwork, but fall short of its architectural elements, which are crucial for integrating the artwork into the viewer’s surrounding space and enabling immersive spatial alignment. This study proposes a spatial augmentation approach that takes an artwork image as input, generates floor patterns and architectural 3D models through 2D and 3D model generation pipelines, and semantically aligns and augments them in the physical space. This spatial augmentation intends to enhance viewer engagement and perceived presence by maintaining spatial coherence between virtual and real space. By activating architectural surfaces as part of the experience, the system repositions AR exhibitions as spatial narratives rather than conventional overlay-driven displays.

IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct) 2025

We propose a lifestyle quantification method for personalized augmented reality (AR) intervention in everyday life. Maintaining a healthy lifestyle accompanies extensive workforce and costs, because accurate health evaluation and tailored management is a prerequisite. However, research on assessing multiple dimensions of health and timely administration of personalized information has been lacking. To address these issues, first we deduced participant’s physical activity, nutrition, interpersonal relations, and stress management level through their mobile devices. Each participant’s week-long data were filtered by clinical features evidenced in high-level medical literature. Next, based on their health status, we presented personalized AR interventions in their simulated everyday environment. Statistical analysis and user experiment results verified the clinical capability of our method and its effect on health-promoting decision behaviors of participants. Through this feasibility study, we hope to contribute on tailored lifestyle management in our daily lives, for better health of all.

IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct) 2025

We present a fast and efficient method for transferring facial textures onto SMPL-X–based full-body avatars. Unlike conventional affine-transform methods that are slow and prone to visual artifacts, our method utilizes a barycentric UV conversion technique. Our approach precomputes the entire UV mapping into a single transformation matrix, enabling texture transfer in a single operation. This results in a speedup of over 7000x compared to the baseline, while also significantly improving the final texture quality by eliminating boundary artifacts. Through quantitative and qualitative evaluations, we demonstrate that our method offers a practical solution for personalization in immersive XR applications. The code is available online.1

IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct) 2025

We present a multimodal framework that uses few-shot language models to generate action scene graphs from raw video for structuring spatial experiences in extended reality (XR). The system integrates unsupervised action segmentation, multimodal captioning with a vision-language model (VLM) guided by speech and motion cues, and scene graph generation from captions using a large language model (LLM), both leveraging few-shot prompting. By combining visual, verbal, and motion inputs, the framework produces semantically rich and accurate representations of spatial experiences without requiring annotated data or model fine-tuning. The resulting outputs will support a range of downstream XR applications, including adaptive playback, intent-driven retrieval, and flexible content reconstruction.

IEEE International Symposium on Mixed and Augmented Reality (ISMAR) 2025Top-tier

As immersive virtual environments become more prevalent, avatars serve as critical social interfaces. This study explores how combinations of visual appearance, vocal characteristics, and informed identity influence users' initial impressions and partner preferences in four distinct XR scenarios: physical, intellectual, social, and romantic. A within-subject experiment with 40 participants assessed perceived warmth, competence, uncanniness, and selection preferences across diverse avatar configurations. Results indicate that vocal cues had a particularly strong impact on social perception, often shaping feelings of approachability and clarity in communication. While some cue alignments enhanced perceived social comfort and engagement, inconsistencies across gender-related cues occasionally led to increased perceptions of uncanniness, especially in emotionally sensitive contexts. These findings highlight the importance of designing avatars that thoughtfully adapt to different interaction contexts, supporting inclusive and responsive user experiences in social XR platforms.

IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct) 2025

This paper presents a comprehensive framework for standardizing avatar face representation in Extended Reality (XR)-based communication. It defines a consistent representation model for facial features and expressions to ensure high visual fidelity and expressive accuracy across diverse XR devices. The proposed framework supports interoperability across different systems and applications by defining core data structures that adapt to the hardware constraints and capabilities of different setups. A key focus is the real-time synchronization of facial expressions, even when users’ faces are occluded by XR devices, to enhance social presence and interaction quality. To address variability in user devices and network conditions, the framework introduces an adaptive control mechanism for real-time avatar rendering. This mechanism dynamically adjusts the visual complexity and expressiveness of avatars based on device performance and network bandwidth, ensuring consistent and responsive communication experiences in heterogeneous environments. Ultimately, this standardization effort aims to advance immersive communication by enabling lifelike, responsive avatars that reflect users’ expressions, thereby accelerating the adoption of XR technologies in both professional and social domains.

IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct) 2025

This paper proposes a mutual space representation to support remote collaboration in Mixed and Augmented Reality (MAR) environments. To achieve high immersion and coexistence in MAR remote collaboration, a mutual space based on heterogeneous physical spaces that can be commonly utilized by collaborating participants is required. Although previous studies have proposed various spatial matching and modifying methods to overcome spatial heterogeneity, their different representations have made their integrated use in real-world scenarios difficult. To overcome these limitations, this paper proposes a mutual space representation standard that includes the representation of physical space, matching space, and collaboration context required for mutual space generation and management. We expect that the proposed standards will serve as a foundation for seamlessly applying solutions accumulated in academia to industrial systems, thereby providing users with a more immersive and coexistent MAR remote collaboration experience.

IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct) 2025

We propose a system that provides personalized interpretation of physical paintings using an AR cursor in a mixed reality (MR) environment. In galleries, pointing at specific regions of complex paintings is often restricted, making it hard to ask targeted questions. While prior works have proposed interactive systems to assist visitor understanding, they typically offered generalized explanations that were not grounded in real, physically exhibited artworks. To address this gap, our approach allows users to intuitively drag across the surface of a painting using a hand-guided AR cursor to specify a Region of Interest. The cropped image, along with the user’s spoken question, is then sent to a Vision Language Model (VLM), which generates a personalized and context-aware explanation based on the selected area. This approach enables users to gain a more precise and meaningful understanding of the painting, guided by their specific interests.

IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct) 2025

We propose Round-Trip2Gesture, an interaction system that uses smartwatch IMU sensors to enable inplace gesture input free from the sensing FOV of devices. Current smartglasses input is limited by social unsuitability of voice commands and by the restricted sensing field-of-view inherent to hand tracking. Our system introduces a delimiter gesture based on specific wrist motions to achieve reliable interaction in real-world scenarios. Our system enables intuitive gesture interaction free from occulsion and physical constraints. We evaluated a set of 5 gestures with 8 participants. Four gestures showed over 90% accuracy in leave-one-subject-out cross-validation. We introduce the potential of the system through notification dismissal and video playback control applications, demonstrating seamless out-of-FOV interaction in Mixed Reality environments.

IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct) 2025

We propose a sequential context engineering framework for zero-shot recognition of procedural tasks in egocentric AR. While large vision-language models excel at zero-shot image classification, they lack temporal reasoning required for procedural understanding, and fully fine-tuning them degrades generalization capabilities. The proposed method preserves frozen CLIP encoders while introducing lightweight modules: an intra-action attention module for temporal aggregation and an inter-action context module that leverages previous predictions to generate context-aware prompts. Evaluated on EGTEA Gaze+ dataset, our approach shows promising results with notable improvements over baselines while maintaining parameter efficiency. This work presents a step toward enabling temporal reasoning for VLMs in resource-constrained AR systems without sacrificing zero-shot generalization.

IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct) 2025

We propose the Spatio-Temporal Mixed and Augmented Reality Experience Description (MAR-ED), a novel framework to standardize the representation of past events for interactive and adaptive playback in a user’s present physical space. While current spatial media technologies have primarily focused on capturing or replaying content as static assets, often disconnected from the viewer’s environment or offering limited interactivity, the means to describe an experience’s underlying semantic and interactive structure remains underexplored. We propose a descriptive framework called MARED based on three core primitives: 1) Event Primitives for semantic scene graph representation, 2) Keyframe Primitives for efficient and meaningful data access, and 3) Playback Primitives for user-driven adaptive interactive playback of recorded MAR experience. The proposed flowchart of the three-stage process of the proposed MAR-ED framework transforms a recorded experience into a unique adaptive MAR experience during playback, where its spatio-temporal structure dynamically conforms to a new environment and its narrative can be altered by live user input. Drawing on this framework, personal digital memories and recorded events can evolve beyond passive 2D/3D videos into immersive, spatially-integrated group experiences, opening new paradigms for training, cultural heritage, and interactive storytelling without requiring complex, per-user adaptive rendering.

Adjunct Proceedings of the 38th Annual ACM Symposium on User Interface Software and Technology

Figure 1: RealityCrafter creates an editable 3D scene driven by user speech from a single real-world image.Our system (a) takes a single image as input; (b) reconstructs it into a 3D scene; and (c) empowers users to author that space via voice commands and pointer interactions.

Proceedings of the 2025 CHI Conference on Human Factors in Computing SystemsTop-tier

This research investigates the use of hybrid user interfaces to enhance text readability in augmented reality (AR) by combining optical see-through head-mounted displays with smartphones. While this integration can improve information legibility, it may also introduce display switching side effects. The extent to which these side effects hinder user experience and when the benefits outweigh drawbacks remain unclear. To address this gap, we conducted an empirical study (N=24) to evaluate how hybrid user interfaces affect AR reading tasks across different content distances, which induce varying levels of display switching. Our findings show that hybrid user interfaces offer significant readability benefits compared to using the HMD only, reducing mental and physical demands when reading text linked to content at closer distances. However, as the distance between displays increases, the compensatory behaviors users adopt to manage increased switching costs negate these benefits, making hybrid user interfaces less effective. Based on these findings, we suggest (1) using smartphones as supplementary displays for text in reading-intensive tasks, (2) implementing adaptive display positioning to minimize switching overhead in such scenarios, and (3) adjusting the smartphone’s role based on content distance for less intensive reading tasks. These insights provide guidance for optimizing smartphone integration in hybrid interfaces and enhancing AR systems for reading applications.

IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW) 2025

Asynchronous communication is essential in diverse contexts such as remote meetings, education, and collaboration. However, traditional replay-based systems lack interactivity and often fail to preserve critical contextual cues. This paper introduces Avagent, a novel framework for asynchronous com munication in Virtual/Augmented Reality (VR/AR) environments that leverages agentized avatars to bridge temporal and contextual gaps between participants. Unlike typical agent-mediated systems, the framework utilizes users’ data to create Avagents that reflect their actual intent, emotional states, and behaviors. By accurately capturing and replicating both verbal and nonverbal behaviors, Avagents enable more interactive and context-rich communication. This approach facilitates dynamic interactions, enhances social presence, and ensures continuity across timelines. Avagents empower new users to engage interactively with prior discussions, fostering deeper understanding and seamless collaboration with previous users. Envisioning the benefits of Avagents as an engaging and context-rich solution for asynchronous communication, this paper outlines interaction scenarios, a work-in-progress prototype, and associated challenges.

IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW) 2025

We present a new approach using Virtual Reality to capture human-scene interaction data within 3D environments. Traditional methods required costly physical setups with real furniture, making it difficult to collect both interaction and 3D scene data comprehensively. Our solution implements a user-centric scene graph with a hierarchical structure that captures both user interactions and 3D scene elements. The resulting dataset contains real human interaction data, dynamic scene information, synthetic sensor data, and scene graph annotations. This work provides a foundation for developing Mixed Reality experiences using 3D scene graphs with human interaction data.

한국소프트웨어종합학술대회 (KSC 2025)

IEEE Transactions on Visualization and Computer GraphicsTop 10%

Immersive technologies such as virtual and augmented reality (VR/AR) allow remote users to meet and interact in a shared virtual space using embodied virtual avatars, creating a sense of co-presence. However, asynchronous communication-essential in many real-world contexts-remains underexplored in these environments. Traditional playback-based systems lack interactivity and often fail to preserve critical contextual cues necessary for effective asynchronous communication. In this paper, we introduce AVAGENTs, AI-powered virtual avatars that replicate users' verbal and nonverbal cues from recordings of past meetings. Avagents can interpret meeting context and generate appropriate responses to questions posed by asynchronous viewers. Through a user study (N = 30), we evaluated Avagents against a traditional playback method and a voice-based AI assistant across two asynchronous meeting scenarios: analytic reasoning and affective resonance. Results showed that Avagents enhance the asynchronous communication experience by increasing social presence, sense of belonging, emotional intimacy, and other user perceptions. We discuss the findings and their implications for designing effective AI-driven asynchronous communication tools in VR/AR environments.

IEEE Transactions on Visualization and Computer Graphics (Proc. IEEE VR'25)Top 10%

As avatars have evolved from simple digital representations into extensions of our identities, they offer unprecedented opportunities for self-expression and customization beyond the physical world limitations. While virtual platforms foster new forms of identity exploration, social norms still play a crucial role in defining what is considered appropriate in these environments. In this study, we surveyed 150 participants to investigate social norms surrounding avatar modifications, examining how perspectives, contexts, and personality traits influence attitudes toward appropriateness. Our findings reveal that avatar modifications are generally viewed as more appropriate when considered from a partner's perspective, especially for changeable attributes. However, these modifications are perceived as less acceptable in professional settings such as workplaces. Additionally, individuals with high self-monitoring tendencies tend to be more resistant to changes, while those scoring higher on Machiavellianism are more accepting of changes, particularly regarding unchangeable attributes and emotional expressions. These findings provide valuable insights for platform developers and designers, highlighting the importance of implementing context-aware customization options that balance core identity elements with personality-driven preferences, thereby enhancing user experiences while respecting social norms.

IEEE Computer Graphics and Applications

We introduce the concept of a meta-object, a next-generation virtual object that inherits the form, properties, and functions of its real-world counterpart, enabling seamless synchronization, interaction, and sharing between the physical and virtual worlds. While plenty of today's virtual objects provide some sensory feedback and dynamic behavior, meta-objects fully integrate interactive and multisensory features within a structured data framework to enable real-time immersive experiences in a postmetaverse intelligent simulation platform. Three key components underpin the utilization of meta-objects in the postmetaverse: property-embedded modeling for physical and action realism, adaptive multisensory feedback tailored to user interactions, and a scene graph-based intelligence simulation platform for scalable and efficient ecosystem integration. By leveraging meta-objects through wearable augmented reality/virtual reality devices, the postmetaverse facilitates seamless interactions that transcend spatial and temporal barriers, paving the way for a transformative reality-virtuality convergence.

GMCV Workshop at CVPR'25 (CVPRW)

Generating high-quality 3D models from 2D sketches is a challenging task due to the inherent ambiguity and sparsity of sketch data. In this paper, we present S3D, a novel framework that converts simple hand-drawn sketches into detailed 3D models. Our method utilizes a U-Net-based encoder-decoder architecture to convert sketches into face segmentation masks, which are then used to generate a 3D representation that can be rendered from novel views. To ensure robust consistency between the sketch domain and the 3D output, we introduce a novel style-alignment loss that aligns the U-Net bottleneck features with the initial encoder outputs of the 3D generation module, significantly enhancing reconstruction fidelity. To further enhance the network's robustness, we apply augmentation techniques to the sketch dataset. This streamlined framework demonstrates the effectiveness of S3D in generating high-quality 3D models from sketch inputs. The source code for this project is publicly available at https://github.com/hailsong/S3D.