Meta-Object

We propose event-based vibrotactile feedback as an effective approach for enhancing bare-hand XR interaction, delivered through commodity smartwatches that users already wear. While controllers and haptic gloves provide rich tactile feedback, they introduce additional hardware that users must carry and wear throughout the day. Importantly, they interpose hardware between the user and the virtual world, occupying the hands, constraining finger motion, and undermining truly unobstructed bare-hand interaction. Our proposed approach delivers short discrete pulses at key manipulation moments such as object contact and state changes, providing tactile confirmation at the wrist without requiring additional hardware or disrupting finger movement. Two user studies with 26 participants demonstrate that this referred feedback significantly enhances user experience at either wrist placement, with 94% finding it helpful and particular benefits during complex tasks like knob rotation where haptic cues reduced visual attention demands. These findings establish design guidelines for integrating accessible haptics into everyday XR through personal devices, supporting broader adoption of natural hand interaction.

We propose a novel object-mediated hand interaction system that enables real-time operation with everyday objects on wearable augmented reality (AR) devices. Despite recent advances, both commercial and academic hand interaction techniques remain constrained, typically requiring external hardware or depending exclusively on bare-hand gestures. Motivated by these constraints, we developed an offloading framework that integrates a high-fidelity transformer-based 3D hand reconstruction model with a dynamic gesture recognition network powered by gated recurrent units (GRU). This architecture ensures stable and accurate gesture recognition even during interaction with physical objects. To evaluate its quantitative performance, we collected a custom dataset based on a predefined gesture set, achieving 93.0% accuracy in 5-fold cross-validation. The complete system implemented on Microsoft HoloLens 2 operates at a real-time framerate, and we further analyze the latency of each step in our framework. Through this interaction paradigm, users can experience immersive and intuitive AR in everyday environments with minimal disruption to natural action behavior. Our projects are available at https://github.com/kaist-uvrlab/UnifiedHOInteraction.

We present ForceCtrl, a novel 3D hand raycasting technique that enhances pointing precision based on control-display (CD) gain controlled with user-defined pinch force. We introduce a target-agnostic approach for refining raycasting precision, overcoming limitations in human motor accuracy. User-defined pinch force, detected with surface electromyography (sEMG), enables users to easily activate or deactivate CD gain during interaction. We propose three CD gain strategies and compare them through target selection and placement tasks. Our system reduces selection errors, placement jitters, and user workload, especially for distant targets in high-difficulty tasks. These results highlight the effectiveness of applying CD gain to hand raycasting and demonstrate the potential of user-defined pinch force as a robust input modality for precise hand interaction in AR/VR.

We propose Round-Trip2Gesture, an interaction system that uses smartwatch IMU sensors to enable inplace gesture input free from the sensing FOV of devices. Current smartglasses input is limited by social unsuitability of voice commands and by the restricted sensing field-of-view inherent to hand tracking. Our system introduces a delimiter gesture based on specific wrist motions to achieve reliable interaction in real-world scenarios. Our system enables intuitive gesture interaction free from occulsion and physical constraints. We evaluated a set of 5 gestures with 8 participants. Four gestures showed over 90% accuracy in leave-one-subject-out cross-validation. We introduce the potential of the system through notification dismissal and video playback control applications, demonstrating seamless out-of-FOV interaction in Mixed Reality environments.

We introduce “Whirling Interface,” a selection method for XR displays using bare-hand motion matching gestures as an input technique. We extend the motion matching input method, by introducing different input states to provide visual feedback and guidance to the users. Using the wrist joint as the primary input modality, our technique reduces user fatigue and improves performance while selecting small and distant targets. In a study with 16 participants, we compared the whirling interface with a standard ray casting method using hand gestures. The results demonstrate that the Whirling Interface consistently achieves high success rates, especially for distant targets, averaging 95.58% with a completion time of 5.58 seconds. Notably, it requires a smaller camera sensing field of view of only 21.45° horizontally and 24.7° vertically. Participants reported lower workloads on distant conditions and expressed a higher preference for the Whirling Interface in general. These findings suggest that the Whirling Interface could be a useful alternative input method for XR displays with a small camera sensing FOV or when interacting with small targets.