Skip to content
KAISTARRCARRC

Papers & Patents

Publications & Patents

264 outputs

202220

Dooyoung Kim* · Jinwook Kim · Jaeeun Shin · Boram Yoon · Jeongmi Lee · Woontack Woo†

IEEE Conference on Virtual Reality and 3D User Interfaces (VR) 2022Top-tier

This paper investigates how the size of virtual space and objects within it affect the threshold range of relative translation gains, a Redirected Walking (RDW) technique that scales the user’s movement in virtual space in different ratios for the width and depth. While previous studies assert that a virtual room’s size affects relative translation gain thresholds on account of the virtual horizon’s location, additional research is needed to explore this assumption through a structured approach to visual perception in Virtual Reality (VR). We estimate the relative translation gain thresholds in six spatial conditions configured by three room sizes and the presence of virtual objects (3 × 2), which were set according to differing Angles of Declination (AoDs) between eye-gaze and the forward-gaze. Results show that both size and virtual objects significantly affect the threshold range, it being greater in the large-sized condition and furnished condition. This indicates that the effect of relative translation gains can be further increased by constructing a perceived virtual movable space that is even larger than the adjusted virtual movable space and placing objects in it. Our study can be applied to adjust virtual spaces in synchronizing heterogeneous spaces without coordinate distortion where real and virtual objects can be leveraged to create realistic mutual spaces.

Hyunjin Lee* · Woontack Woo†

IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW) 2022

Providing additional information in the proper position of augmented reality (AR) head-mounted display (HMD) can help increase AR performance and usability for dual-task. Therefore, our study inves-tigated how to place notifications for the dual-task to address this. We compared eight display positions and two tasks (single and dual tasks) to identify the appropriate area for displaying notifications. We confirmed that the middle-right reduces response time and task load. In contrast, the top-left is the location, which should avoid providing any notification in AR dual-task. Our study contributes to designing AR notifications on HMDs to enhance everyday AR experiences.

Hyerim Park* · Maryam Shakeri · Ikbeom Jeon · Jangyoon Kim · Abolghasem Sadeghi-Niaraki · Woontack Woo†

Springer Virtual Reality

There have been attempts to provide new cinematic experiences by connecting TV or movie content to suitable locations through augmented reality (AR). However, few studies have suggested a method to manage breakdowns in continuity due to spatial transitions. Thus, we propose a method to manage the spatial transition that occurs when we create a TV show trajectory by mapping TV show scenes with spatiotemporal information to the real world. Our approach involved two steps. The first step is to reduce the spatial transition considering the sequence, location, and importance of TV show scenes when creating the TV show trajectory in the authoring tool. The second is to fill the spatial transition with additional TV show scenes considering sequence, importance, and user interest when providing the TV show trajectory in the mobile application. The user study results showed that reducing spatial transition increases narrative engagement by allowing participants to see important content within the trajectory. The additional content in spatial transition decreased the physical demand and effort in terms of the perceived workload, although it increased the task completion time. Integrated spatial transition management improved the overall cinematic augmented reality (CAR) experience of the TV show. Furthermore, we suggest design implications for realizing the CAR of TV shows based on our findings.

CONVERSATIONAL DIGITAL TWIN METHOD USING GEOGRAPHIC INFORMATION AND APPARATUS THEREOFPat.
우운택 · 신승엽

Other · Pending

Disclosed are a conversational digital twin method and apparatus using geographic information. A conversational digital twin method using geographic information, which is performed by a computer device, includes obtaining geographic information data converted into 2D or 3D; visualizing the geographic information data in a terminal and providing visualized information to a user; and interacting with the terminal of the user through an artificial intelligence or a voice recognition-based chatbot using a conversational interface to manipulate the visualized information or control an external system.

METHOD AND SYSTEM FOR SYNTHESIZING NOVEL VIEW IMAGE ON BASIS OF MULTIPLE 360 IMAGES FOR 6-DEGREES OF FREEDOM VIRTUAL REALITYPat.
Woontack Woo · Hochul Cho

Other · Pending

A method and a system for synthesizing a novel-view image based on multiple 360 images for 6DOF virtual reality, in which a large-scale 6-DOF virtual environment is implemented, and a scene is synthesized at a novel viewpoint, are provided. The method includes performing a 3D reconfiguration procedure for the 360 images to recover 3D geometric information, and to reconfigure a virtual data map in which the multiple 360 images are integrated into one image, producing a view image corresponding to a viewpoint of a user by applying a view synthesis algorithm of projection & vertex warping process using a reference image which is closest to a viewpoint extracted from the virtual data map, and bl

Jaeeun Shin* · Boram Yoon · Dooyoung Kim · Woontack Woo†

IEEE Transactions on Visualization and Computer GraphicsTop 10%

A critical yet unresolved challenge in designing space-adaptive narratives for Augmented Reality (AR) is to provide consistently immersive user experiences anywhere, regardless of physical features specific to a space. For this, we present a comprehensive analysis on a series of user studies investigating how the size, density, and layout of real indoor spaces affect users playing Fragments, a space-adaptive AR detective game. Based on the studies, we assert that moderate levels of traversability and visual complexity afforded in counteracting combinations of size and complexity are beneficial for narrative experience. To confirm our argument, we combined the experimental data of the studies (n=112) to compare how five different spatial complexity conditions impact narrative experience when applied to contrasting room sizes. Results show that whereas factors of narrative experience are rated significantly higher in relatively simple settings for a small space, they are less affected by complexity in a large space. Ultimately, we establish guidelines on the design and placement of space-adaptive augmentations in location-independent AR narratives to compensate for the lack or excess of affordances in various real spaces and enhance user experiences therein.

물체와 상호작용하는 손의 추적을 위한 맨손 깊이 인페인팅 방법 및 시스템Pat.
우운택 · 조우진 · 박갑용

KR 10-2395123-0000 · Korea · Granted

본 발명은 조건부 적대적 생성망(cGAN)으로 구현되는 맨손 깊이 인페인팅을 사용하여 물체와 상호작용하는 손을 3D 추적하는 손의 추적을 위한 맨손 깊이 인페인팅 방법 및 시스템에 관한 것으로, 물체와 상호작용하는 손에 대한 카메라 이미지를 수신하는 단계, 상기 카메라 이미지에서 손 및 물체를 추적하여 깊이 인페인팅(depth inpainting)을 이용한 맨손 깊이 이미지를 생성하는 단계 및 상기 맨손 깊이 이미지에서 손 포즈를 추정하는 단계를 포함한다.

실제 및 가상 사물인터넷 기기를 연동한 3차원 디지털 트윈 시각화 시스템 및 그 방법Pat.
우운택 · 신승엽 · 조호철

KR 10-2441454-0000 · Korea · Granted

본 발명은 실제 및 가상 사물인터넷 기기를 연동한 3차원 디지털 트윈 시각화 시스템 및 그 방법에 관한 것으로서, 실세계에 배치되는 실제 사물인터넷(IoT) 기기로부터 실제 지리공간 데이터를 수집하며, 가상세계에서 서버의 셀프 루틴을 통해 생성되는 가상 사물인터넷(IoT) 기기로부터 가상 지리공간 데이터를 수집하는 가상 및 실제 데이터 수집부, 상기 실제 지리공간 데이터 및 상기 가상 지리공간 데이터를 통합하여 도시 디지털 트윈(Urban Digital Twin; UDT)에 전송하는 데이터 처리부 및 상기 통합된 데이터를 기반으로 상호작용을 위한 증강현실의 동시 공간을 제공하며, 테이블탑(Tabletop)으로부터 제어를 위해 생성된 네트워크 패킷을 중계하는 도시 디지털 트윈부를 포함한다.

Yun Suen Pai* · Marsel L. Bait · Juyoung Lee · Jingjing Xu · Roshan L Peiris · Woontack Woo · Mark Billinghurst · Kai Kunze†

Springer Virtual Reality

We present NapWell, a Sleep Assistant using virtual reality (VR) to decrease sleep onset latency by providing a realistic imagery distraction prior to sleep onset. Our proposed prototype was built using commercial hardware and with relatively low cost, making it replicable for future works as well as paving the way for more low cost EOG-VR devices for sleep assistance. We conducted a user study (n = 20) by comparing different sleep conditions; no devices, sleeping mask, VR environment of the study room and preferred VR environment by the participant. During this period, we recorded the electrooculography (EOG) signal and sleep onset time using a finger tapping task (FTT). We found that VR was able to significantly decrease sleep onset latency. We also developed a machine learning model based on EOG signals that can predict sleep onset with a cross-validated accuracy of 70.03%. The presented study demonstrates the feasibility of VR to be used as a tool to decrease sleep onset latency, as well as the use of embedded EOG sensors with VR for automatic sleep detection.

202119

Hyung-il Kim* · Taehei Kim · Eunhwa Song · Seo Young Oh · Dooyoung Kim · Woontack Woo†

IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct) 2021

In this poster, we present a digital twin-based mixed reality system for remote collaboration with the size-scaling of the user and the space. The proposed system supports collaboration between an AR host user and a VR remote user by sharing a 3D digital twin of the AR host user. To enhance the coarse authoring of a shared digital twin environment, we provide a size scaling of the digital twin environment with the world-in-miniature view. Also, we enable scaling the size of the VR user’s avatar to enhance both coarse (size-up) and fine-grained (size-down) authoring of the digital twin environment. We describe the system setup, input methods, and interaction methods for scaling space and user.

Gi-Bbeum Lee* · Hyuckjin Jang · Hyundeok Jeong · Woontack Woo†

IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct) 2021

In remote collaboration using Augmented Reality (AR), speech and gesture are major communication methods for the general public. However, the Deaf and Hard-of-Hearing (DHH) population cannot join in the communication due to the absence of a sign language interface which is their primary language. Recent works have tried to augment spoken language with sign language animations or captions, but the research to convey sign language with spoken language is still very limited. In this paper, we propose a novel multi-modal communication system that integrates sign language translation, speech recognition, and shared object manipulation in the mobile AR environment. Though the system is currently under development, we demonstrated a rapid prototype of the telemedicine app leveraging the video prototyping method to integrate the system modules. We performed preliminary interviews about our approach with DHH users, a sign language interpreter, and a physician. We discuss the insights into the future design of the DHH communication support in the AR collaboration system. This study has a socio-cultural, economic impact on the DHH population as a barrier-free design of a remote collaboration system in a practical scenario. Another contribution of this work is that we suggested a novel user-centered system for DHH users in AR by integrating the existing technologies.

Seungyoub Ssin* · Hochul Cho · Woontack Woo†

Interaction Design and Architecture(s)

Despite prior efforts to solve urban problems and build smart cities using large-scale urban information, researchers have yet to develop tools for city administrators to easily access a diverse mix of city information. To address this limitation, this article proposes an augmented reality visualization system, GeoACT, which can develop a digital twin to present city information from both macro and micro perspectives. GeoACT employs a 3D virtual city miniature with real and virtual IoT information to provide a macro perspective view, and a 360-degree-video-based visualization technique is used for a micro perspective view. We applied GeoACT to an urban space and demonstrated that it could effectively visualize city information through an augmented reality Head Mounted Display (HMD). Furthermore, GeoACT can be used in urban control centers.

Jaeeun Shin* · Boram Yoon · Dooyoung Kim · Woontack Woo†

Proceedings of the 2021 CHI Conference on Human Factors in Computing SystemsTop-tier

Space-adaptive algorithms aim to effectively align the virtual with the real to provide immersive user experiences for Augmented Reality(AR) content across various physical spaces. While such measures are reliant on real spatial features, efforts to understand those features from the user’s perspective and reflect them in designing adaptive augmented spaces have been lacking. For this, we compared factors of narrative experience in six spatial conditions during the gameplay of Fragments, a space-adaptive AR detective game. Configured by size and furniture layout, each condition afforded disparate degrees of traversability and visibility. Results show that whereas centered furniture clusters are suitable for higher presence in sufficiently large rooms, the same layout leads to lower narrative engagement. Based on our findings, we suggest guidelines that can enhance the effects of space adaptivity by considering how users perceive and navigate augmented space generated from different physical environments.

Hui-Shyong Yeo* · Juyoung Lee · Woontack Woo · Hideki Koike · Aaron J Quigley · Kai Kunze†

Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems

In this work, we explore a new sensing technique for smart eyewear equipped with Electrooculography (EOG) sensors. We repurpose the EOG sensors embedded in a JINS MEME smart eyewear, originally designed to detect eye movement, to detect midair hand gestures. We also explore the potential of sensing human proximity, rubbing action and to differentiate materials and objects using this sensor. This new found sensing capabilities enable a various types of novel input and interaction scenarios for such wearable eyewear device, whether it is worn on body or resting on a desk.

Seungyoub Ssin* · Hochul Cho · Woontack Woo†

Augmented Reality and Virtual Reality

This paper introduces a method for developing Augmented Urban Digital Twin (A-UDT) for virtual and real Internet-of-Things (IoT) visualization and its benefits. It presents a method for generating virtual data using the Virtual IoT (VI) based on real IoT data, which can be used with cloud services. An urban digital twin is developed to find and simulate various problems arising in the city by extracting and visualizing data of the city. Designing urban digital twin has some challenging issues such as the difficulty of processing environmental information in real-time due to the absence of IoT support. This absence of IoT support can make problems like developing urban digital twin without knowing the types and scope of visualization. To overcome this limitation, we propose the augmented city miniature that uses generation system, process, storage, and network system of the component that generates VI with the same data structure as real IoT to generate and send data, and it simulates information on a 3D geographic map by sending the generated virtual data to the visualization part using Augmented Reality (AR). Urban digital twin based on the virtual data can make various simulations of city visualization that can be performed to discover required IoT and reduce development costs. A-UDT system can be applied to monitor and simulate diverse environmental information such as air quality, energy efficiency, object movements, and temperature distribution.

Seungyoub Ssin* · Minjeong Suh · Jongwook Lee · Timothy Jung · Woontack Woo†

Augmented Reality and Virtual Reality

The purpose of this study is to propose a theoretical framework for a digital twin-based smart science tourism system using augmented reality in Gwanghwamun Square. To date, smart tourism service was not sufficiently developed for the provision of relevant tourism information to visitors as tourism contents were managed by the separated information system of each institution. The tourists could not receive ideal tourism routes based on their preferences and also according to congestion of each institution. Moreover, the business model using existing tourism services was not well developed, and thus the current tourism services have not sufficiently contributed to the revitalization of the regional economy. We designed a digital twin-based tourism content management system that provides integrated management of tourism information, and augmented reality (AR) science tour services which are included with contents recommendation, best path finding, contents sharing and gamification. We also adopted a business model using the AR tourism platform based on digital twin technology which could contribute to the invigoration of the local economy.

Dooyoung Kim* · Jaeeun Shin · Jeongmi Lee · Woontack Woo†

IEEE Virtual Reality and 3D User Interfaces (VR) 2021

We propose the concept of relative translation gains, a novel Redirected Walking (RDW) method to create a mutual movable space between the Augmented Reality (AR) host's reference space and the Virtual Reality (VR) client's space. Previous RDW methods have focused on maximizing the movable space at the expense of aligning the coordinates between the AR and VR side, and could only be applied to collaborative scenarios involving sequential tasks. Our method solves these problems by adjusting the remote client's walking speed for each axis of a VR space to modify the movable area without coordinate distortion. We estimate the relative translation gain threshold, defined as the extent to which the walking speed can be altered without creating a perceived difference in distance. In order to reflect features of the reference space in generating the mutual space, we then examine how changing its size affects the threshold value. Our study showed that for remote clients connected to the larger reference space, relative translation gains can be increased to utilize a VR space bigger than their real space. Our method can be applied to create optimal mutual spaces for a wider variety of asymmetric Mixed Reality (MR) remote collaboration systems.

Hyerim Park* · Woontack Woo†

IEEE Virtual Reality and 3D User Interfaces (VR) 2021

Most research on providing location-based content in 3D interactive virtual reality has been limited to social media content. Few studies have suggested how to represent the video clip of movies or TV shows in virtual reality. This paper investigates a video content representation method to provide a hyper-reality experience of the narrative world in virtual reality. We reflect the time and place settings of the video content in virtual reality and have participants watch the video in four different virtual reality environments. We reveal that reflecting the story's environment settings to the virtual reality environment significantly improves the spatial presence and narratives engagement. We also confirm a positive correlation between spatial presence and narrative engagement, including subscales such as emotional engagement and narrative presence. Based on the study results, we discuss how to provide the hyper-reality experience in content-adaptive virtual reality.

Inhwa Yeom* · Woontack Woo†

IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW) 2021

We present a digital twin-based Mixed Reality platform for arts curators. Despite the rising presence of exhibitions in varying digital formats, the virtual engagement of arts curators has been nearly disabled. To replicate and enhance the professional capability of arts curators in the virtual realm, our system integrates a digital twin art space and 3D authoring techniques, aided with spatial and semantic metadata. Our user evaluation proves the system's usability, and its capacity to support curatorial activities. With this, we aim to provide a groundwork to similar systems that also extend arts curator's creativity and outreach beyond time and space.

Seungyoub Ssin* · Hochul Cho · Woontack Woo†

IEEE International Conference on Consumer Electronics (ICCE) 2021

This paper shows a structure and implementation of an Urban Digital Twin (UDT), simulating virtual data productions using Virtual IoT (VI) to design services for citizens. When designing smart city services with UDT systems, developing the system without data support from real IoTs can be created without knowing the scope and limitations of the IoTs. We simulated the services by creating VI system, which can be introduced in the future based on the meaningful data sets of existing Real IoT sensors. This paper focuses on the concept of the proposed system, GeoVCM, the components that make up the system.

HAND POSE ESTIMATION METHOD WITH A SINGLE INFRARED CAMERA VIA DOMAIN TRANSFER LEARNING AND THE SYSTEM THEREOFPat.
우운택 · 박갑용 · 김태균

PCT/KR2021/095031 · PCT · Pending

The present invention relates to a hand position estimation method and system for estimating a 3-dimensional hand position for quick hand movement, on the basis of domain transfer learning from a depth image to an infrared image, the hand position estimation method comprising the steps of: processing a depth image and an infrared image for hand movement; synthesizing a depth map with the infrared image by using a hand image generator (HIG), and estimating skeletal positions of hand joints from each of the depth map and an infrared map; and calculating a 3-dimensional hand position by using the skeletal positions and the center of a hand depth image.

Jinwoo Park* · Ikbeom Jeon · Sung-Eui Yoon · Woontack Woo†

IEEE Transactions on Visualization and Computer GraphicsTop 10%

증강현실 기반 인터렉티브 저작 서비스 제공 시스템Pat.
우운택 · 길경원 · 하태진 · 도영임 · 임지민

KR 10-2304023-0000 · Korea · Granted

본 발명은 HMD(Head Mounted Display)가 구비된 웨어러블(wearable) 디바이스와, 상기 웨어러블 디바이스와 페어링(pairing)되어 GUI 인터페이스를 통해 시나리오 기반 기설정된 플로우(flow)에 대응하는 콘텐츠를 재생하고, 상기 콘텐츠에 형성된 오브젝트에 인터럽트(interrupt) 발생 시 상기 웨어러블 디바이스로부터 뷰잉(viewing) 되는 3차원 공간에 해당 오브젝트를 오버레이(overlay)하여 증강현실 영상을 생성하고, 사용자 제스처(gesture)에 따라 오버레이된 오브젝트별 상태를 변환하고, 움직임 센서를 통해 센싱되는 움직임 정보를 기반으로 상기 오브젝트의 위치 영역을 변환하는 증강현실 서비스 제공 단말과, 자기 센서(magnetic sensor)를 구비하고, 상기 증강현실 서비스 제공 단말로부터 출력되는 오브젝트를 선택하거나 활성화하는 포인팅 디바이스를 포함함을 특징으로 한다.

증강현실을 이용한 기타 운지 학습 시스템Pat.
우운택 · 도승원 · 김형일

KR 10-2228019-0000 · Korea · Granted

본원의 일 측면에 따른 증강현실을 이용한 기타 운지 학습 시스템은 기타 영상을 촬영하는 카메라 모듈; 입력 모듈; 증강현실 영상이 출력되는 디스플레이 모듈; 증강현실을 이용한 기타 운지 학습 프로그램이 탑재된 메모리; 및 프로그램을 실행하는 프로세서를 포함한다. 기타는 복수의 스트링을 포함하고 지판의 일측에 증강현실 마커가 부착된 것이고, 프로세서는 기타 운지 학습 프로그램의 실행에 따라, 기타 영상에서 증강현실 마커를 인식하고, 증강현실 마커를 기초로 지판 상의 스트링 영역 및 프렛 영역을 추출하여 원본 이미지로 저장하고, 입력 모듈을 통해 입력 받은 악보 파일에서 곡의 박자, 운지 또는 주법이 포함된 연주 정보를 추출하고, 추출된 연주 정보를 이용하여 지판에서 운지해야 하는 위치 및 탄현해야 하는 스트링을 알려주는 표준 운지 이미지를 생성하고, 표준 운지 이미지와 원본 이미지를 중첩시킨 증강현실 영상을 생성하고, 생성된 증강현실 영상을 디스플레이 모듈을 통해 출력하는 것을 특징으로 한다.

한국HCI학회 학술대회 발표 논문집, 2021

Hyunju Kim* · Sanghwa Hong · Junki Kim · Taesoo Jang · Woontack Woo · Seongkook Heo · Byungjoo Lee†

Multimedia Tools and Applications

This study introduces RealityBrush, a novel augmented reality (AR) authoring system that allows designers to quickly and easily create realistic virtual objects by capturing and utilizing the kinetic properties of everyday physical objects in the early stages of design. The RealityBrush system consists of a handheld device, a data analysis module and an AR feedback module. The handheld device, which is made in the shape of a rod, is equipped with a depth camera and a force sensor at the tip. When a user holds the device and pokes a physical object, the local force applied to the object and the resulting deformations of the object are measured simultaneously. By analyzing the relationship between measured force and deformations, the RealityBrush system can identify two kinetic properties of the poked object: stiffness and motion resistance. The user can then use the handheld device as a 3D brush to create a virtual object in the air and assign the measured kinetic properties to the created virtual object. Finally, the system’s physics engine allows the user to interact with the created object by using the device to poke or push the object. The technical evaluation showed that the system can successfully extract the stiffness and motion resistance of everyday objects. We also report initial user feedback on AR authoring using the RealityBrush system.

202015

Gabyong Park* · Tae-Kyun Kim · Woontack Woo†

IEEE International Symposium on Mixed and Augmented Reality (ISMAR) 2020Top-tier

Previous methods successfully estimated 3D hand poses from unblurred depth images with slow and smooth hand motions. However, the performance drops when the depth images are contaminated by motion blur due to fast hand motion. In this paper, we exploit an infrared (IR) image input, which is weakly blurred under fast hand motion. The proposed method is based on domain transfer learning from depth to infrared images. Note we do not have IR images with hand skeletons, thus proposing self-supervision rather than direct supervision using the skeleton labels. We train a Hand Image Generator (HIG) and two Hand Pose Estimators (HPEs) on paired depth and infrared images via self-supervision using a consistency loss, guided by an existing HPE trained on paired depth and hand skeleton entries. The IR-based HPE is then refined on the weakly blurred infrared images. The qualitative and quantitative experiments demonstrate that the proposed method accurately estimates 3D hand poses under motion blur by fast hand motion, while existing depth-based methods fail. Our solution therefore supports fast 3D manipulation of virtual objects for augmented reality applications. Our model and dataset are publicly available for future research.1

Woojin Cho* · Gabyong Park · Woontack Woo†

IEEE International Symposium on Mixed and Augmented Reality (ISMAR) 2020Top-tier

We propose a 3D hand tracking system using bare-hand depth inpainting from an RGB-depth image for a hand interacting with an object. The effectiveness of most existing hand-object tracking methods is impeded by the insufficiency of data, which do not include hand data occluded by the object, and their reliance on the information inferred from assuming the specific object type. We generate a sufficiently accurate bare-hand depth image from a hand interacting with an object using a conditional generative adversarial network, which is trained using the synthesized 2D silhouettes of the object to learn the morphology of the hand. We evaluate the proposed approach using a hierarchical particle filter-based hand tracker and prove that our approach utilizing the bare-hand tracker in the hand-object interaction dataset achieve state-of-the-art performance. The generalization of our work will enable visual-tactile interaction that is more natural in various wearable augmented reality applications.

Sunyoung Bang* · Hyunjin Lee · Woontack Woo†

IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct) 2020

Using an augmented reality head-mounted display(HMD) to extend the display of smartphone could open up new user interface solutions, benefiting users with an increased screen real-estate and visualizations that leverage each device's capabilities. Some previous works have explored the viability of extended displays, but knowledge regarding considerable design factors and constraints for such displays is still very limited. In our work, we conducted an exploratory study to investigate how different properties of augmented content in extended displays affect the user's task performance and subjective workload in an image comparison task. Through an experiment with 24 participants, we compared four augmented content placements (top, right, left, and bottom) and two augmented content sizes (small and large). The study results demonstrated that there are significant effects of both placement and size of augmented content on task performance and subjective workload. Based on the findings of our study, we propose some recommendations for the future design of extended displays.

Hyunjin Lee* · Sunyoung Bang · Woontack Woo†

IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct) 2020

Information in augmented reality (AR) consists of virtual and real contexts and is provided to the AR environment at different distances for each user. Therefore, it is important to understand how the integration of two different information influences the user's AR experience. However, little has been studied regarding the complexity and viewing distance of a real space. Our study investigates how the complexity of the physical environment and viewing distance influence workload and performance in a visual search task in an AR environment. We conducted an experiment in which participants performed conjunction search under three different levels of background complexity, at both near (1.5m) and far (3m) distances. The results indicated that as the complexity of the background increased, the users' performance time and workload were negatively impacted. In addition, when the distance between the user and the background was greater, search time increased. From the results of the study, we derived some recommendations for the design of AR interfaces. Our research contributes to the design of interfaces by demonstrating the necessity to consider the complexity of background and viewing distance.

Boram Yoon* · Hyung-il Kim · Seo Young Oh · Woontack Woo†

IEEE International Symposium on Mixed and Augmented Reality (ISMAR) 2020Top-tier

This study investigates the effects of a virtual hand representation on the user experience including social presence during hand-based 3D remote collaboration. Although a remote hand appearance is a critical parts of a hand-based telepresence, it has been rarely studied in comparison to studies on the self-embodiment of virtual hands in a 3D environment. Thus, we conducted a user study comparing the three virtual hands models (Skeleton, Low Polygon and Realistic) while performing a remote collaborative task based on the American Sign Language (ASL) using both Augmented Reality (AR) and Virtual Reality (VR) environments. We found that the realistic type was perceived as the most sense of being together, human-like, and trustable representation. The low polygon model could also convey a clear sign and moderate level of social presence. Although the system was configured asymmetrically in AR and VR, little difference in perception was found except for the participant's mental load and message understanding. We then discuss the results and suggest design implications for future hand-based 3D telepresence systems.

Seo Young Oh* · Boram Yoon · Hyung-il Kim · Woontack Woo†

Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems

This paper reports that the time-domain accuracy of bare-hand interactions in HMD-based Augmented Reality can be improved by using finger contact: touching a finger with another or tapping one's own hand. The activation of input can be precisely defined by the moment of finger contact, allowing the user to perform the input precisely at the desired moment. Finger contact is better suited to the user's mental model, and natural tactile feedback from the fingertip also benefits the user with the self-perception of the input. The experimental results revealed that using finger contact is the preferred method of input that increases the time-domain accuracy and enables the user to be aware of the moment the input is activated.

Richard Li* · Juyoung Lee · Woontack Woo · Thad Starner†

Proceedings of the Augmented Humans International Conference

Cheek kissing is a common greeting in many countries around the world. Many parameters are involved when performing the kiss, such as which side to begin the kiss on and how many times the kiss is performed. These parameters can be used to infer one's social and physical context. In this paper, we present KissGlass, a system that leverages off-the-shelf smart glasses to recognize different kinds of cheek kissing gestures. Using a dataset we collected with 5 participants performing 10 gestures, our system obtains 83.0% accuracy in 10-fold cross validation and 74.33% accuracy in a leave-one-user-out user independent evaluation.

Hui-Shyong Yeo* · Juyoung Lee · Andrea Bianchi · Alejandro Samboy · Hideki Koike · Woontack Woo · Aaron Quigley†

Proceedings of the Augmented Humans International Conference

WristLens is a system for surface interaction from wrist-worn wearable devices such as smartwatches and fitness trackers. It enables eyes-free, single-handed gestures on surfaces, using an optical motion sensor embedded in a wrist-strap. This allows the user to leverage any proximate surface, including their own body, for input and interaction. An experimental study was conducted to measure the performance of gesture interaction on three different body parts. Our results show that directional gestures are accurately recognized but less so for shape gestures. Finally, we explore the interaction design space enabled by WristLens, and demonstrate novel use cases and applications, such as on-body interaction, bimanual interaction, cursor control and 3D measurement.

360 카메라 및 디지털 트윈을 기반으로 한 이동 객체의 지리적 데이터 추정방법 및 시스템Pat.
우운택 · 조호철 · 신승엽

Application No. 10-2020-0177977 · Korea · Pending

본 발명은 360 카메라 및 디지털 트윈을 기반으로 한 이동 객체의 지리적 데이터 추정방법 및 시스템에 관한 것으로, 360 카메라에서 영상 데이터를 수신하여 전처리하는 단계, 상기 영상 데이터 내 객체의 위치를 추출하며, 디지털 트윈(Digital Twin)을 통해 이미지 깊이(depth)를 추정하여 상기 객체의 초기 위치를 추정하는 단계 및 상기 초기 위치를 기반으로 상기 디지털 트윈에 배치된 3D 모델(3D model)을 이용하여 상기 영상 데이터 내 상기 객체와의 차이를 계산하며, 차이를 최소화하는 최적화(optimization)를 통해 상기 객체의 위치 및 각도를 포함하는 지리적 데이터를 추정하는 단계를 포함한다.

Gabyong Park* · Antonis Argyros · Juyoung Lee · Woontack Woo†

IEEE Transactions on Visualization and Computer GraphicsTop 10%

We present a sensor-fusion method that exploits a depth camera and a gyroscope to track the articulation of a hand in the presence of excessive motion blur. In case of slow and smooth hand motions, the existing methods estimate the hand pose fairly accurately and robustly, despite challenges due to the high dimensionality of the problem, self-occlusions, uniform appearance of hand parts, etc. However, the accuracy of hand pose estimation drops considerably for fast-moving hands because the depth image is severely distorted due to motion blur. Moreover, when hands move fast, the actual hand pose is far from the one estimated in the previous frame, therefore the assumption of temporal continuity on which tracking methods rely, is not valid. In this paper, we track fast-moving hands with the combination of a gyroscope and a depth camera. As a first step, we calibrate a depth camera and a gyroscope attached to a hand so as to identify their time and pose offsets. Following that, we fuse the rotation information of the calibrated gyroscope with model-based hierarchical particle filter tracking. A series of quantitative and qualitative experiments demonstrate that the proposed method performs more accurately and robustly in the presence of motion blur, when compared to state of the art algorithms, especially in the case of very fast hand rotations.

NOVEL VIEW SYNTHESIS METHOD BASED ON MULTIPLE 360 IMAGES FOR 6-DOF VIRTUAL REALITY AND THE SYSTEM THEREOFPat.
우운택 · 조호철 · 김장윤

PCT/KR2020/002289 · PCT · Pending

The present invention relates to a method and a system for synthesizing a novel view image on the basis of multiple 360 images for 6-degrees of freedom (DoF) virtual reality, wherein multiple 360 images are used to construct a large-scale 6-DoF virtual environment and synthesize a scene at a new viewpoint. The method comprises the steps of: performing a 3D reconfiguration procedure on 360 images to reconstruct 3D geometric information and reconfigure a virtual data map integrating the multiple 360 images into one image; using a reference image closest to a viewpoint extracted from the virtual data map to apply a projection 0026# vertex warping view synthesis algorithm so as to create view im

Jinwoo Park* · Hunmin Park · Sung-Eui Yoon · Woontack Woo†

IEEE Transactions on Visualization and Computer GraphicsTop 10%

In mixed reality (MR), augmenting virtual objects consistently with real-world illumination is one of the key factors that provide a realistic and immersive user experience. For this purpose, we propose a novel deep learning-based method to estimate high dynamic range (HDR) illumination from a single RGB image of a reference object. To obtain illumination of a current scene, previous approaches inserted a special camera in that scene, which may interfere with user's immersion, or they analyzed reflected radiances from a passive light probe with a specific type of materials or a known shape. The proposed method does not require any additional gadgets or strong prior cues, and aims to predict illumination from a single image of an observed object with a wide range of homogeneous materials and shapes. To effectively solve this ill-posed inverse rendering problem, three sequential deep neural networks are employed based on a physically-inspired design. These networks perform end-to-end regression to gradually decrease dependency on the material and shape. To cover various conditions, the proposed networks are trained on a large synthetic dataset generated by physically-based rendering. Finally, the reconstructed HDR illumination enables realistic image-based lighting of virtual objects in MR. Experimental results demonstrate the effectiveness of this approach compared against state-of-the-art methods. The paper also suggests some interesting MR applications in indoor and outdoor scenes.

듀얼 카메라가 장착된 이동형 디바이스를 이용한 포커스 콘텍스트 영상 처리 장치 및 그 방법Pat.
우운택 · 김기홍 · 김장윤

KR 10-2074072-0000 · Korea · Granted

본 발명은 듀얼 카메라가 장착된 이동형 디바이스를 이용한 포커스 콘텍스트 영상 처리 장치 및 그 방법에 관한 것으로, 이동형 디바이스의 고화질 카메라로부터 촬영되는 주요 관심 영역의 고화질 영상 데이터, 및 광각 카메라로부터 촬영되는 주변 영역의 광각 영상 데이터를 이용하여 증강현실 영상 콘텐츠를 제공할 수 있다.

손 자세 인식 및 적응적 공간 탐색을 활용한 손 관절 추적 방법 및 그 장치Pat.
우운택 · 박갑용

KR 10-2150794-0000 · Korea · Granted

손 자세 인식 및 적응적 공간 탐색을 활용한 손 관절 추적 방법 및 그 장치가 개시된다. 본 발명의 일 실시예에 따른 손 자세 추적 방법은 손 자세를 포함하는 깊이 이미지를 수신하는 단계; 상기 깊이 이미지에 기초하여 상기 손 자세에 대한 제1 이미지와 제2 이미지를 생성하는 단계; 상기 제1 이미지에 기초하여 상기 제1 이미지에 대응하는 손 대표 자세를 분류하는 단계; 상기 제2 이미지와 상기 분류된 손 대표 자세에 기초하여 미리 설정된 파티클들 각각을 생성하는 단계; 상기 생성된 파티클들 각각에 대한 깊이 오류와 자세 오류를 계산하는 단계; 상기 계산된 깊이 오류와 자세 오류를 기반으로 상기 파티클들 각각과 검색 공간 범위를 업데이트하는 단계; 및 상기 업데이트된 파티클들 각각과 검색 공간 범위에 기초하여 손 자세를 추정하는 단계를 포함한다.

한국HCI학회 학술대회 발표 논문집, pp.1080-1083, 2020

201915

Jaeeun Shin* · Hayun Kim · Callum Parker · Hyung-il Kim · Seoyoung Oh · Woontack Woo†

IEEE International Symposium on Mixed and Augmented Reality (ISMAR) 2019Top-tier

One of the main challenges in creating narrative-driven Augmented Reality (AR) content for Head Mounted Displays (HMDs) is to make them equally accessible and enjoyable in different types of indoor environments. However, little has been studied in regards to whether such content can indeed provide similar, if not the same, levels of experience across different spaces. To gain more understanding towards this issue, we examine the effect of room size and furniture on the player experience of Fragments, a space-adaptive, indoor AR crime-solving game created for the Microsoft HoloLens. The study compares factors of player experience in four types of spatial conditions: (1) Large Room - Fully Furnished; (2) Large Room - Scarcely Furnished; (3) Small Room - Fully Furnished; and (4) Small Room - Scarcely Furnished. Our results show that while large spaces facilitate a higher sense of presence and narrative engagement, fully-furnished rooms raise perceived workload. Based on our findings, we propose design suggestions that can support narrative-driven, space-adaptive indoor HMD-based AR content in delivering optimal experiences for various types of rooms.

Shuo Liu* · Seogsung Jang · Woontack Woo†

IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct) 2019

Augmented reality is a technology which extends measureless contents in the virtual space and makes actual objects not limited in the physical properties. As the most popular cultural media among the campus, the poster have its shortcoming of offering restricted information. In this paper, we introduce a responsive poster-reading system which generates diverse contents step by step with body-based and hand-based interactions, and a design guideline which applies for corresponding posters. The proposed method is able to provide users with a user-friendly, immersive and intuitive experience.

Juyoung Lee* · Shaurye Aggarwal · Jason Wu · Thad Starner · Woontack Woo†

Proceedings of the 23rd International Symposium on Wearable Computers

SelfSync enables rapid, robust initiation of a gesture interface using synchronized movement of different body parts. SelfSync is the gestural equivalent of a hotword such as OK-Google in a speech interface and is enabled by the increasing trend where a user wears two or more wearables, such as a smartwatch, wireless earbuds, or a smartphone. In a user study comparing five potential SelfSync gestures in isolation, our system averages 96%, 98% and 88% for user dependent, user adapted, and user independent accuracy, respectively. For when the user has a phone in a pocket and a smart-watch, we suggest twisting the hand about the wrist while moving the leg with the phone in synchrony left and right. When the user has a head worn device and a smartwatch, we suggest twisting the hand while twisting the head left and right.

Seungwon Kim* · Gun Lee · Weidong Huang · Hayun Kim · Woontack Woo · Mark Billinghurst†

Proceedings of the 2019 CHI Conference on Human Factors in Computing SystemsTop-tier

Many researchers have studied various visual communication cues (e.g. pointer, sketching, and hand gesture) in Mixed Reality remote collaboration systems for real-world tasks. However, the effect of combining them has not been so well explored. We studied the effect of these cues in four combinations: hand only, hand + pointer, hand + sketch, and hand + pointer + sketch, with three problem tasks: Lego, Tangram, and Origami. The study results showed that the participants completed the task significantly faster and felt a significantly higher level of usability when the sketch cue is added to the hand gesture cue, but not with adding the pointer cue. Participants also preferred the combinations including hand and sketch cues over the other combinations. However, using additional cues (pointer or sketch) increased the perceived mental effort and did not improve the feeling of co-presence. We discuss the implications of these results and future research directions.

Hochul Cho* · Jangyoon Kim · Woontack Woo†

IEEE Conference on Virtual Reality and 3D User Interfaces (VR) 2019Top-tier

We present a novel view synthesis method that allows users to experience a large-scale Six-Degree-of-Freedom (6-DOF) virtual environment. Our main contributions are the construction of a large-scale 6-DOF virtual environment using multiple 360 images as well as synthesis of a scene from novel viewpoints. Novel view synthesis from a single 360 image can give free viewpoint experience with full 6-DOF of head motion to players, but the moveable space is limited within a context of the image. We propose a novel view synthesis process that references multiple 360 images via reconstructing a large-scale real world based virtual data map and perform a weighted blending for interpolating multiple novel view images. Our results show that our approach provides a wider area of virtual environment as well as a smooth transition between each reference 360 images.