Selin Choi

We propose a method for generating task breakpoints based on an Origin-Centric Graph (OCG) to segment goal-oriented activity recordings into task units for adaptive playback in Virtual Reality (VR) environments. With the development of Augmented Reality (AR)/VR head-mounted displays (HMDs), research on adaptive tutorials and authoring tools has become active, but existing task segmentation methods mainly rely on manual annotation or are restricted to 2D video which limits their applicability to 3D VR contexts. In our approach, assembly scenarios with clearly defined task boundaries are recorded using a structured spatio-temporal scene graph (STSG), and the OCG is employed to track changes in the central object and the formation of new groups, thereby generating task breakpoints automatically. A user study collected user-perceived task breakpoints to establish ground truth (GT), and comparison with the algorithm-detected breakpoints demonstrated high agreement and confirmed accuracy in supporting adaptive playback. The proposed task segmentation method provides a foundation for dynamically adjusting VR playback according to user proficiency and progress, with potential for extension into automatic timeline segmentation systems for diverse VR recordings.

We present object-driven event retrieval through an assembled object interface for part-specific navigation in Mixed Reality (MR) assembly playback. Users may need to recheck how two visible parts attach, inspect where subassemblies merge, or locate the step involving parts they are handling, yet time-based navigation requires mapping these part-relation questions to timeline positions, while prior object-based approaches mainly support single-object statechange retrieval. Our system couples part selection on the completed assembly with retrieval over inter-part relations encoded in a scene-graph-based recording, enabling direct access to the relevant joining process through objects rather than timeline search. In a within-subjects experiment with 24 participants across three assembly scenarios, the Object Interface (OB) significantly reduced task completion time relative to the breakpoint-augmented timeline, showed scenario-dependent advantages over the plain timeline, and was selected in the vast majority of preference responses as the most suitable interface for part-specific lookup. These results support object-driven retrieval as a complementary navigation mode for targeted lookup when users know the relevant parts.

We propose a spatial affordance-aware affine transformation method between heterogeneous spaces for continuous multi-object matching in shared Mixed Reality (MR) spaces. While previous redirection and spatial mapping approaches utilize physical objects and walkable areas, a critical gap remains in enabling continuous mapping between dissimilar physical environments that supports both precise object alignment and seamless locomotion in a shared space. Our method structurally segments heterogeneous spaces into interaction zones and constructs affine patches based on object adjacency and facing configuration, enabling continuous correspondence. We evaluate our method using a dataset of paired dissimilar spaces and demonstrate that, unlike conventional grid-based methods, our approach achieves broader spatial alignment and richer object matching. The results show that our method can serve as an effective mapping framework for shared environments requiring semantic continuity and structural coherence across diverse real-world spaces.

We present a new approach using Virtual Reality to capture human-scene interaction data within 3D environments. Traditional methods required costly physical setups with real furniture, making it difficult to collect both interaction and 3D scene data comprehensively. Our solution implements a user-centric scene graph with a hierarchical structure that captures both user interactions and 3D scene elements. The resulting dataset contains real human interaction data, dynamic scene information, synthetic sensor data, and scene graph annotations. This work provides a foundation for developing Mixed Reality experiences using 3D scene graphs with human interaction data.

To enable remote Virtual Reality (VR) and Augmented Reality (AR) clients to collaborate as if they were in the same space during Mixed Reality (MR) telepresence, it is essential to overcome spatial heterogeneity and generate a unified shared collaborative environment by integrating remote spaces into a target host space. Especially when multiple remote users connect, a large shared space is necessary for people to maintain their personal space while collaborating, but the existing simple intersection method leads to the creation of narrow shared spaces as the number of remote spaces increases. To robustly align to the host space even as the number of remote spaces increases, we propose a spatial affordance-aware interactable subspace allocation algorithm. The key concept of our approach is to consider the perceivable and interactable areas separately, where every user views the same mutual space, but each remote user has a different interactable subspace, considering their location and spatial affordance. We conducted an evaluation with 900 space combinations, varying the number of remote spaces as two, four, and six, and results show our method outperformed in securing wide interactable mutual space and instantiating users compared to the other spatial matching methods. Our work enables multiple clients from diverse remote locations to access the AR host’s space, allowing them to interact directly with the table, wall, or floor by aligning their physical subspaces within a connected mutual space.

We introduce the concept of a meta-object, a next-generation virtual object that inherits the form, properties, and functions of its real-world counterpart, enabling seamless synchronization, interaction, and sharing between the physical and virtual worlds. While plenty of today's virtual objects provide some sensory feedback and dynamic behavior, meta-objects fully integrate interactive and multisensory features within a structured data framework to enable real-time immersive experiences in a postmetaverse intelligent simulation platform. Three key components underpin the utilization of meta-objects in the postmetaverse: property-embedded modeling for physical and action realism, adaptive multisensory feedback tailored to user interactions, and a scene graph-based intelligence simulation platform for scalable and efficient ecosystem integration. By leveraging meta-objects through wearable augmented reality/virtual reality devices, the postmetaverse facilitates seamless interactions that transcend spatial and temporal barriers, paving the way for a transformative reality-virtuality convergence.