Taewook Ha

Development of Open XR platform for high immersive collaboration

We propose Int3DNet, a scene-aware network that predicts 3D intention areas directly from scene geometry and head-hand motion cues, enabling robust human intention prediction without explicit object-level perception. In Mixed Reality (MR), intention prediction is critical as it enables the system to anticipate user actions and respond proactively, reducing interaction delays and ensuring seamless user experiences. Our method employs a cross attention fusion of sparse motion cues and scene point clouds, offering a novel approach that directly interprets the user's spatial intention within the scene. We evaluated Int3DNet on MoGaze and CIRCLE datasets, which are public datasets for full-body human-scene interactions, showing consistent performance across time horizons of up to 1500 ms and outperforming the baselines, even in diverse and unseen scenes. Moreover, we demonstrate the usability of proposed method through a demonstration of efficient visual question answering (VQA) based on intention areas. Int3DNet provides reliable 3D intention areas derived from head-hand motion and scene geometry, thus enabling seamless interaction between humans and MR systems through proactive processing of intention areas.

We propose a novel crowd data-driven optimization approach for artwork placement in virtual exhibitions. With the emerging concept of Metaverse, a multitude of users can engage with content contemporaneously in virtual exhibitions. Yet, few studies have suggested a method to resolve crowd density concentration in multiuser Mixed Reality (MR) and Virtual Reality (VR) environments. In this study, our approach leverages crowd data engaged with artworks to predict optimal placement of artworks to distribute crowd density in virtual exhibitions, prior to exhibition planning. To investigate the requirement and validity of our approach, we conducted focus group interviews and an artwork relocation experiment as preliminary studies. In the generation of solution scenes for optimal placement, our optimizer adaptively scrutinizes placeable areas with considerations of crowd density distribution, scene rationality, and artwork similarity. Through a performance comparison analysis between optimization results, we confirmed that the optimizer successfully fulfilled the intended objectives with respect to the design considerations, resolving practical scenarios in exhibition planning.

We present a new approach using Virtual Reality to capture human-scene interaction data within 3D environments. Traditional methods required costly physical setups with real furniture, making it difficult to collect both interaction and 3D scene data comprehensively. Our solution implements a user-centric scene graph with a hierarchical structure that captures both user interactions and 3D scene elements. The resulting dataset contains real human interaction data, dynamic scene information, synthetic sensor data, and scene graph annotations. This work provides a foundation for developing Mixed Reality experiences using 3D scene graphs with human interaction data.

We propose a novel object-mediated hand interaction system that enables real-time operation with everyday objects on wearable augmented reality (AR) devices. Despite recent advances, both commercial and academic hand interaction techniques remain constrained, typically requiring external hardware or depending exclusively on bare-hand gestures. Motivated by these constraints, we developed an offloading framework that integrates a high-fidelity transformer-based 3D hand reconstruction model with a dynamic gesture recognition network powered by gated recurrent units (GRU). This architecture ensures stable and accurate gesture recognition even during interaction with physical objects. To evaluate its quantitative performance, we collected a custom dataset based on a predefined gesture set, achieving 93.0% accuracy in 5-fold cross-validation. The complete system implemented on Microsoft HoloLens 2 operates at a real-time framerate, and we further analyze the latency of each step in our framework. Through this interaction paradigm, users can experience immersive and intuitive AR in everyday environments with minimal disruption to natural action behavior. Our projects are available at https://github.com/kaist-uvrlab/UnifiedHOInteraction.