Meta-Object

We propose event-based vibrotactile feedback as an effective approach for enhancing bare-hand XR interaction, delivered through commodity smartwatches that users already wear. While controllers and haptic gloves provide rich tactile feedback, they introduce additional hardware that users must carry and wear throughout the day. Importantly, they interpose hardware between the user and the virtual world, occupying the hands, constraining finger motion, and undermining truly unobstructed bare-hand interaction. Our proposed approach delivers short discrete pulses at key manipulation moments such as object contact and state changes, providing tactile confirmation at the wrist without requiring additional hardware or disrupting finger movement. Two user studies with 26 participants demonstrate that this referred feedback significantly enhances user experience at either wrist placement, with 94% finding it helpful and particular benefits during complex tasks like knob rotation where haptic cues reduced visual attention demands. These findings establish design guidelines for integrating accessible haptics into everyday XR through personal devices, supporting broader adoption of natural hand interaction.

We propose a novel object-mediated hand interaction system that enables real-time operation with everyday objects on wearable augmented reality (AR) devices. Despite recent advances, both commercial and academic hand interaction techniques remain constrained, typically requiring external hardware or depending exclusively on bare-hand gestures. Motivated by these constraints, we developed an offloading framework that integrates a high-fidelity transformer-based 3D hand reconstruction model with a dynamic gesture recognition network powered by gated recurrent units (GRU). This architecture ensures stable and accurate gesture recognition even during interaction with physical objects. To evaluate its quantitative performance, we collected a custom dataset based on a predefined gesture set, achieving 93.0% accuracy in 5-fold cross-validation. The complete system implemented on Microsoft HoloLens 2 operates at a real-time framerate, and we further analyze the latency of each step in our framework. Through this interaction paradigm, users can experience immersive and intuitive AR in everyday environments with minimal disruption to natural action behavior. Our projects are available at https://github.com/kaist-uvrlab/UnifiedHOInteraction.

Question-asking is one of the key indicators of cognitive engagement. However, understanding how the distinct psychological affordances of presentation media shape learners’ spoken inquiries with embodied Intelligent Virtual Agents (IVAs) remains limited. To systematically examine this process, we propose a 5W1H-based framework for analyzing learner questions. Using this framework, we conducted a user study comparing an Augmented Reality-based IVA (AR-IVA) deployed in the physical environment with a screen-based IVA (Video-IVA) during cardiopulmonary resuscitation (CPR) instruction. Results showed that the AR-IVA elicited higher spatial and social presence and promoted more frequent and longer questions focused on clarification and understanding. In contrast, the Video-IVA encouraged questions regarding procedural refinement. Presence acted as a selective filter, shaping the timing and …

We propose a robust 3D hand tracking system in various hand action environments, including hand-object interaction, which utilizes a single color image and a previous pose prediction as input. We observe that existing methods deterministically exploit temporal information in motion space, failing to address realistic diverse hand motions. Also, prior methods paid less attention to efficiency as well as robust performance, i.e., the balance issues between time and accuracy. The Temporally Enhanced Graph Convolutional Network (TE-GCN) utilizes a 2-stage framework to encode temporal information adaptively. The system establishes balance by adopting an adaptive GCN, which effectively learns the spatial dependency between hand mesh vertices. Furthermore, the system leverages the previous prediction by estimating the relevance across image features through the attention mechanism. The proposed method achieves state-of-the-art balanced performance on challenging benchmarks and demonstrates robust results on various hand motions in real scenes. Moreover, the hand tracking system is integrated into a recent HMD with an off-loading framework, achieving a real-time framerate while maintaining high performance. Our study improves the usability of a high-performance hand-tracking method, which can be generalized to other algorithms and contributes to the usage of HMD in everyday life. Our code with the HMD project will be available at https://github.com/UVR-WJCHO/TEGCN_on_Hololens2 .

We present ForceCtrl, a novel 3D hand raycasting technique that enhances pointing precision based on control-display (CD) gain controlled with user-defined pinch force. We introduce a target-agnostic approach for refining raycasting precision, overcoming limitations in human motor accuracy. User-defined pinch force, detected with surface electromyography (sEMG), enables users to easily activate or deactivate CD gain during interaction. We propose three CD gain strategies and compare them through target selection and placement tasks. Our system reduces selection errors, placement jitters, and user workload, especially for distant targets in high-difficulty tasks. These results highlight the effectiveness of applying CD gain to hand raycasting and demonstrate the potential of user-defined pinch force as a robust input modality for precise hand interaction in AR/VR.

We propose Round-Trip2Gesture, an interaction system that uses smartwatch IMU sensors to enable inplace gesture input free from the sensing FOV of devices. Current smartglasses input is limited by social unsuitability of voice commands and by the restricted sensing field-of-view inherent to hand tracking. Our system introduces a delimiter gesture based on specific wrist motions to achieve reliable interaction in real-world scenarios. Our system enables intuitive gesture interaction free from occulsion and physical constraints. We evaluated a set of 5 gestures with 8 participants. Four gestures showed over 90% accuracy in leave-one-subject-out cross-validation. We introduce the potential of the system through notification dismissal and video playback control applications, demonstrating seamless out-of-FOV interaction in Mixed Reality environments.

We introduce “Whirling Interface,” a selection method for XR displays using bare-hand motion matching gestures as an input technique. We extend the motion matching input method, by introducing different input states to provide visual feedback and guidance to the users. Using the wrist joint as the primary input modality, our technique reduces user fatigue and improves performance while selecting small and distant targets. In a study with 16 participants, we compared the whirling interface with a standard ray casting method using hand gestures. The results demonstrate that the Whirling Interface consistently achieves high success rates, especially for distant targets, averaging 95.58% with a completion time of 5.58 seconds. Notably, it requires a smaller camera sensing field of view of only 21.45° horizontally and 24.7° vertically. Participants reported lower workloads on distant conditions and expressed a higher preference for the Whirling Interface in general. These findings suggest that the Whirling Interface could be a useful alternative input method for XR displays with a small camera sensing FOV or when interacting with small targets.