Hyung-il Kim

In this paper, we present a prototype system for sharing a user's hand force in mixed reality (MR) remote collaboration on physical tasks, where hand force is estimated using wearable surface electromyography (sEMG) sensor. In a remote collaboration between a worker and an expert, hand activity plays a crucial role. However, the force exerted by the worker's hand has not been extensively investigated. Our sEMG-based system reliably captures the worker's hand force during physical tasks and conveys this information to the expert through hand force visualization, overlaid on the worker's view or on the worker's avatar. A user study was conducted to evaluate the impact of visualizing a worker's hand force on collaboration, employing three distinct visualization methods across two view modes. Our findings demonstrate that sensing and sharing hand force in MR remote collaboration improves the expert's awareness of the worker's task, significantly enhances the expert's perception of the collaborator's hand force and the weight of the interacting object, and promotes a heightened sense of social presence for the expert. Based on the findings, we provide design implications for future mixed reality remote collaboration systems that incorporate hand force sensing and visualization.

In this poster, we present a digital twin-based mixed reality system for remote collaboration with the size-scaling of the user and the space. The proposed system supports collaboration between an AR host user and a VR remote user by sharing a 3D digital twin of the AR host user. To enhance the coarse authoring of a shared digital twin environment, we provide a size scaling of the digital twin environment with the world-in-miniature view. Also, we enable scaling the size of the VR user’s avatar to enhance both coarse (size-up) and fine-grained (size-down) authoring of the digital twin environment. We describe the system setup, input methods, and interaction methods for scaling space and user.

In this demonstration, we will show a prototype system with sensor fusion approach to robustly track 6 degrees of freedom of hand movement and support intuitive hand gesture interaction and 3D object manipulation for Mixed Reality head-mounted displays. Robust tracking of hand and finger with egocentric camera remains a challenging problem, especially with self-occlusion - for example, when user tries to grab a virtual object in midair by closing the palm. Our approach leverages the use of a common smart watch worn on the wrist to provide a more reliable palm and wrist orientation data, while fusing the data with camera to achieve robust hand motion and orientation for interaction.

We introduce a smartwatch assisted sensor fusion approach to robustly track 6-DOF hand movement in head mounted display (HMD) based augmented reality (AR) environment, which can be used for robust 3D object manipulation. Our method uses a wrist-worn smartwatch with HMD-mounted depth sensor to robustly track 3D position and orientation of user's hand. We introduce HMD-based augmented reality platform with smartwatch, and method to accurately calibrate orientation between smartwatch and HMD. We also implement natural 3D object manipulating system using 6-DOF hand tracker with hand grasping detection. Our proposed system is easy to use, and doesn't require any hand held devices.

An omni-directional (360°) camera captures the entire viewing sphere surrounding its optical center. Such cameras are growing in use to create highly immersive content and viewing experiences. When such a camera is held by a user, the view includes the user’s hand grip, finger, body pose, face, and the surrounding environment, providing a complete understanding of the visual world and context around it. This capability opens up numerous possibilities for rich mobile input sensing. In OmniSense, we explore the broad input design space for mobile devices with a built-in omni-directional camera and broadly categorize them into three sensing pillars: i) near device ii) around device and iii) surrounding device. In addition we explore potential use cases and applications that leverage these sensing capabilities to solve user needs. Following this, we develop a working system to put these concepts into action, by leveraging these sensing capabilities to enable potential use cases and applications. We studied the system in a technical evaluation and a preliminary user study to gain initial feedback and insights. Collectively these techniques illustrate how a single, omni-purpose sensor on a mobile device affords many compelling ways to enable expressive input, while also affording a broad range of novel applications that improve user experience during mobile interaction.

This study proposed a Mixed Reality (MR) mutual space gener-ation method that aligns the Virtual Reality (VR) clients' spaces around the Augmented Reality (AR) host space's reference object and adjusts the virtual space's scale with translation-based Redi-rected Walking (RDW). Although there were methods to create shared spaces by finding optimal transform or each space to max-imize the semantically aligned area, there was a lack of research on utilizing the effect of rescaling a VR client's space with the RDW technique. We leveraged Relative Translation Gains (RTGs), a length-based RDW technique, to maximize the semantic information matching ratio between the AR host space and the VR clients' spaces by rescaling the client's virtual space within a suitable threshold range according to the spatial configuration. Our pilot study results show that the semantic information matching rate between the AR and the VR space increased compared to the existing one-to-one scale matching method while maintaining a unified coordinate sys-tem. Our findings can be used to create a more expansive mutual space for asymmetric remote collaboration even if remote spaces increase and challenging to secure a sufficient shared area.