Skip to content
KAISTARRCARRC

Papers & Patents

Publications & Patents

264 outputs

202440

텍스트 읽기가 수반되는 증강현실 작업을 수행하는 증강현실 시스템 및 그 방법Pat.
우운택 · 방선영

KR 10-2728699-0000 · Korea · Granted

본 발명은 단말기를 보조 디스플레이로 활용한 증강현실 헤드마운트 디스플레이 내 읽기 경험을 향상시키는 증강현실 시스템 및 그 방법에 관한 것으로서, 텍스트 읽기가 수반되는 증강현실 작업시 텍스트를 표시하는 단말기 및 텍스트 읽기가 수반되는 증강현실 작업시 텍스트를 분리하여 상기 단말기에 제공하고, 상기 텍스트가 분리된 3차원 객체를 표시하는 증강현실 헤드마운트 디스플레이를 포함한다.

혼합 현실 텔레프레즌스 시스템의 동작 방법 및 이를 수행하는 혼합 현실 텔레프레즌스 시스템Pat.
우운택 · 김두영

Application No. 10-2024-0005235 · Korea · Pending

혼합 현실 텔레프레즌스 시스템의 동작 방법은, 호스트의 제1 가상 공간 및 복수의 클라이언트들의 복수의 제2 가상 공간들로부터 목표 물체가 설정된 제1 부분 공간 및 상기 목표 물체가 설정된 복수의 제2 부분 공간들을 추출하는 단계, 상기 제1 부분 공간, 상기 복수의 제2 부분 공간들 및 최적화 함수에 기초하여, 상기 제1 부분 공간에 상기 복수의 제2 부분 공간들을 오버랩(overlap)한 복수의 정렬 공간들을 생성하는 단계, 상기 복수의 정렬 공간들 상의 복수의 제1 위치들을 결정하는 단계, 및 상기 복수의 제1 위치들에 기초하여, 상기 제1 가상 공간에 복수의 제1 상호작용 가능 영역들을 설정하는 단계를 포함한다.

혼합현실 속 원격 의료를 위해 디지털 트윈을 이용한 전인적 환자 진단 방법 및 그 시스템Pat.
우운택 · 김태연 · 권현송 · 조경현

Application No. 10-2024-0161975 · Korea · Pending

원격 진료 시스템은 제1 사용자의 조작에 의해 상기 제1 사용자에게 제1 확장 현실을 표시하는 제1 확장 현실 장치, 제2 사용자의 조작에 의해 상기 제2 사용자에게 제2 확장 현실을 표시하는 제2 확장 현실 장치 및 상기 제1 확장 현실 및 상기 제2 확장 현실을 제공하기 위한 데이터를 저장하는 데이터 서버를 포함하고, 상기 제1 사용자가 상기 제2 사용자를 원격으로 진단하기 위한 진료실의 환경에 대한 정보를 포함하는 진료실 데이터 및 상기 제2 사용자의 생활 환경에 대한 정보를 포함하는 생활 환경 데이터는 상기 데이터 서버에 저장되며, 상기 제1 확장 현실 장치가 상기 진료실 데이터에 기초하여 상기 제1 확장 현실로서 상기 진료실을 표시할 때, 상기 제1 확장 현실 장치는 상기 진료실 내에서 생활 환경 버튼을 더 표시하고, 상기 제1 사용자에 의해 상기 생활 환경 버튼이 선택될 때, 상기 제1 확장 현실 장치는 상기 생활 환경 데이터에 기초하여 상기 제1 확장 현실로서 상기 제2 사용자의 상기 생활 환경을 표시한다.

한국소프트웨어종합학술대회 (KSC 2024)

KCC2024 Korean Comprehensive Computer Science Conference Paper Collection Vol.2024

202329

Dooyoung Kim* · Woontack Woo†

IEEE International Symposium on Mixed and Augmented Reality (ISMAR) 2023Top-tier

We propose a novel space-rescaling technique for registering dissimilar physical-virtual spaces by utilizing the effects of adjusting physical space with redirected walking. Achieving a seamless and immersive Virtual Reality (VR) experience requires overcoming the spatial heterogeneities between the physical and virtual spaces and accurately aligning the VR environment with the user’s tracked physical space. However, existing space-matching algorithms that rely on one-to-one scale mapping are inadequate when dealing with highly dissimilar physical and virtual spaces, and redirected walking controllers could not utilize basic geometric information from physical space in the virtual space due to coordinate distortion. To address these issues, we apply relative translation gains to partitioned space grids based on the main interactable object’s edge, which enables space-adaptive modification effects of physical space without coordinate distortion. Our evaluation results demonstrate the effectiveness of our algorithm in aligning the main object’s edge, surface, and wall, as well as securing the largest registered area compared to alternative methods under all conditions. These findings can be used to create an immersive play area for VR content where users can receive passive feedback from the plane and edge in their physical environment.

Eunhwa Song* · Taewook Ha · Junhyeok Park · Hyunjin Lee · Woontack Woo†

IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct) 2023

We propose the Holistic Quantified-Self (HQS), a sensor-driven user model for AR glass users. Existing research on context-aware AR interfaces typically relies on singular context attributes, resulting in an incomplete representation of user states. To address this gap, we propose an integrated user model for AR glass users, encompassing physical condition, cognitive-emotional status, social circumstances, and digital consumption behavior. Our system collects data from various devices, enabling simultaneous prediction of user states and tasks within the AR environment. We envision that this approach paves the way for future adaptive AR interfaces that can dynamically adjust virtual content based on the recognized user states and tasks.

Hail Song* · Boram Yoon · Woojin Cho · Woontack Woo†

IEEE International Symposium on Mixed and Augmented Reality (ISMAR) 2023Top-tier

We present a novel method for avatar body generation that cumulatively updates the texture and normal map in real-time. Multiple images or videos have been broadly adopted to create detailed 3D human models that capture more realistic user identities in both Augmented Reality (AR) and Virtual Reality (VR) environments. However, this approach has a higher spatiotemporal cost because it requires a complex camera setup and extensive computational resources. For lightweight reconstruction of personalized avatar bodies, we design a system that progressively captures the texture and normal values using a single RGBD camera to generate the widely-accepted 3D parametric body model, SMPL-X. Quantitatively, our system maintains real-time performance while delivering reconstruction quality comparable to the state-of-the-art method. Moreover, user studies reveal the benefits of real-time avatar creation and its applicability in various collaborative scenarios. By enabling the production of high-fidelity avatars at a lower cost, our method provides more general way to create personalized avatar in AR/VR applications, thereby fostering more expressive self-representation in the metaverse.

Maryam Shakeri* · Hyerim Park · Ikbeom Jeon · Abolghasem Sadeghi-Niaraki · Woontack Woo†

Springer Virtual Reality

There have been studies on personalized augmented reality (AR) systems taking users’ contexts and histories into account. However, there is insufficient research on incorporating real-time user behavior and interactions into the personalized recommendations in spatial transitions, which can be used for new users without user history data. The spatial transitions, distances between two Point of Interests (POIs) of an AR tour trajectory through which users should pass to visit AR contents, need to be filled using the personalized AR contents to reduce the discontinuity experience. This paper aims to propose a user behavior model to recommend personalized contents in the AR tour trajectory to create a personalized experience. First, we model three interactions, including staring, skipping, and liking, using mathematical methods; and two spatial behaviors, including standing and moving, using a rule engine. Second, a content filtering-based recommendation method is presented to apply the proposed user model to recommend personalized AR contents in the spatial transitions. The experiment results showed that the proposed real-time user behavior model brought notable improvements in creating a personal experience of the AR tour system. The proposed method outperformed the conventional method in terms of recommendation performance metrics, so that it achieved an average of 15% more precision and recall scores and an average of 32% more personalization scores than those of the conventional method. Furthermore, the level of personalization perceived by the proposed user model demonstrated significant positive relationships with the level of decision effectiveness, system trustworthiness, and perceived recommendation transparency. The findings showed the effectiveness of the proposed framework in solving the cold-start recommendation problem.

Seoyoung Kang* · Hail Song · Boram Yoon · Kangsoo Kim · Woontack Woo†

IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct) 2023

In this paper, we delve into the influence of different facial blend-shape combinations on social presence and communication quality in avatar-mediated remote communication scenarios: informative speech and emotional conversation. The conducted user study (n = 32) underscores the crucial role of emotion-based blendshapes, suggesting that the emotionally essential blendshape combination outlines levels of social presence and communication quality. Our research highlights the potential of a streamlined approach for avatar facial control, facilitating emotional expressions and optimizing resources, fostering an avatar-mediated communication experience.

Jaeeun Shin* · Woontack Woo†

Proceedings of the 2023 CHI Conference on Human Factors in Computing SystemsTop-tier

We report on a qualitative study in which 22 participants created Augmented Reality (AR) stories for outdoor cultural heritage sites. As storytelling is a crucial strategy for AR content aimed at providing meaningful experiences, the emphasis has been on what storytelling does, rather than how it is done, the end user’s needs prioritized over the author’s. To address this imbalance, we identify how recurring patterns in the spatial trajectories and narrative compositions of AR stories for cultural heritage sites are linked to the author’s intent and creative process: While authors tend to bind story arcs tightly to confined trajectories for narrative delivery, the need for spatial exploration results in thematic content mapped loosely onto encompassing trajectories. Based on our analysis, we present design recommendations for site-specific AR storytelling tools that can support authors in delivering their intent while leveraging the placeness of cultural heritage sites as a creative resource.

Hui-Shyong Yeo* · Erwin Wu · Daehwa Kim · Juyoung Lee · Hyung-il Kim · Seo Young Oh · Luna Takagi · Woontack Woo · Hideki Koike · Aaron John Quigley†

Proceedings of the 2023 CHI Conference on Human Factors in Computing SystemsTop-tier

An omni-directional (360°) camera captures the entire viewing sphere surrounding its optical center. Such cameras are growing in use to create highly immersive content and viewing experiences. When such a camera is held by a user, the view includes the user’s hand grip, finger, body pose, face, and the surrounding environment, providing a complete understanding of the visual world and context around it. This capability opens up numerous possibilities for rich mobile input sensing. In OmniSense, we explore the broad input design space for mobile devices with a built-in omni-directional camera and broadly categorize them into three sensing pillars: i) near device ii) around device and iii) surrounding device. In addition we explore potential use cases and applications that leverage these sensing capabilities to solve user needs. Following this, we develop a working system to put these concepts into action, by leveraging these sensing capabilities to enable potential use cases and applications. We studied the system in a technical evaluation and a preliminary user study to gain initial feedback and insights. Collectively these techniques illustrate how a single, omni-purpose sensor on a mobile device affords many compelling ways to enable expressive input, while also affording a broad range of novel applications that improve user experience during mobile interaction.

Sunyoung Bang* · Woontack Woo†

IEEE Conference on Virtual Reality and 3D User Interfaces (VR) 2023Top-tier

The reading experience on current augmented reality (AR) head mounted displays (HMDs) is often impeded by the devices' low perceived resolution, translucency, and small field of view, especially in situations involving lengthy text. Although many researchers have proposed methods to resolve this issue, the inherent characteristics prevent these displays from delivering a readability on par with that of more traditional displays. As a solution, we explore the use of smartphones as assistive displays to AR HMDs. To validate the feasibility of our approach, we conducted a user study in which we compared a smartphone-assisted hybrid interface against using the HMD only for two different text lengths. The results demonstrate that the hybrid interface yields a lower task load regardless of the text length, although it does not improve task performance. Furthermore, the hybrid interface provides a better experience regarding user comfort, visual fatigue, and perceived readability. Based on these results, we claim that joining the spatial output capabilities of the HMD with the high-resolution display of the smartphone is a viable solution for improving the reading experience in AR.

Hyunjin Lee* · Woontack Woo†

IEEE Conference on Virtual Reality and 3D User Interfaces (VR) 2023Top-tier

Augmented reality (AR) helps users easily accept information when they are walking by providing virtual information in front of their eyes. However, it remains unclear how to present AR notifications considering the expected user reaction to interruption. Therefore, we investigated to confirm appropriate placement methods for each type by dividing it into notification types that are handled immediately (high) or that are performed later (low). We compared two coordinate systems (display-fixed and body-fixed) and three positions (top, right, and bottom) for the notification placement. We found significant effects of notification type and placement on how notifications are perceived during the AR notification experience. Using a display-fixed coordinate system responded faster for high notification types, whereas using a body-fixed coordinate system resulted in quick walking speed for low ones. As for the position, the high types had a higher notification performance at the bottom position, but the low types had enhanced walking performance at the right position. Based on the finding of our experiment, we suggest some recommendations for the future design of AR notification while walking.

Eunhwa Song* · Minju Baeck · Jihyeon Lee · Seo Young Oh · Dooyoung Kim · Woontack Woo · Jeongmi Lee · Sang Ho Yoon†

IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW) 2023

We propose Memo:me, an AR sticky note with priority-based color transition and on-time reminders in smartphones. For the priority-based color transition, the user can choose by himself among three different colors, or the system changes the color automatically 10 minutes before the entered time. For the on-time reminder, Memo:me provides a visual notification and a sound at the designated time. We enabled users to create virtual notes on planes, or carry daily objects with the virtual notes attached. We expect that our system would benefit users to manage their tasks in a time-appropriate manner.

Boram Yoon* · Jaeeun Shin · Hyung-il Kim · Seo Young Oh · Dooyoung Kim · Woontack Woo†

IEEE Transactions on Visualization and Computer Graphics ( Volume: 29, Issue: 11, November 2023)Top 10%

Despite the importance of avatar representation on user experience for Mixed Reality (MR) remote collaboration involving various device environments and large amounts of task-related information, studies on how controlling visual parameters for avatars can benefit users in such situations have been scarce. Thus, we conducted a user study comparing the effects of three avatars with different transparency levels (Nontransparent, Semi-transparent, and Near-transparent) on social presence for users in Augmented Reality (AR) and Virtual Reality (VR) during task-centric MR remote collaboration. Results show that avatars with a strong visual presence are not required in situations where accomplishing the collaborative task is prioritized over social interaction. However, AR users preferred more vivid avatars than VR users. Based on our findings, we suggest guidelines on how different levels of avatar transparency should be applied based on the context of the task and device type for MR remote collaboration.

INFRARED CAMERA-BASED METHOD AND SYSTEM FOR ESTIMATING HAND POSITION THROUGH DOMAIN TRANSFER LEARNINGPat.
우운택 · 박갑용 · 김태균

Application No. 18/259,615 · Other · Pending

The present invention relates to a hand position estimation method and system for estimating a 3-dimensional hand position for quick hand movement, on the basis of domain transfer learning from a depth image to an infrared image, the hand position estimation method comprising the steps of: processing a depth image and an infrared image for hand movement; synthesizing a depth map with the infrared image by using a hand image generator (HIG), and estimating skeletal positions of hand joints from each of the depth map and an infrared map; and calculating a 3-dimensional hand position by using the skeletal positions and the center of a hand depth image.

Woontack Woo

KCC2023 Korean Comprehensive Computer Science Conference Paper Collection Vol.2023

RGB-D 데이터 기반 3차원 재구성 기법을 이용한 공유 공간 생성 및 원격 공유를 위한 시스템, 방법, 및 컴퓨터 프로그램Pat.
우운택 · 신승엽

Application No. 10-2023-0094817 · Korea · Pending

RGB-D 데이터 기반 3차원 재구성 기법을 이용한 공유 공간 생성 및 원격 공유를 위한 시스템, 방법, 및 컴퓨터 프로그램이 개시된다. 공간 기반 원격 XR(eXtended Reality) 협업 시스템은, 현장의 공간 영상으로 RGB-D(depth) 이미지를 제공하는 호스트 단말; 상기 호스트 단말로부터 상기 RGB-D 이미지를 수신하여 저장하는 엣지 컴퓨팅 시스템(Edge Computing System); 및 상기 현장에 대한 원격의 협업자(Collaborator) 디바이스로서 상기 엣지 컴퓨팅 시스템으로부터 상기 RGB-D 이미지를 수신하여 메시 모델(mesh model)로 재구성하고 상기 메시 모델을 렌더링하여 가시화하는 게스트 단말을 포함할 수 있다.

Hyung-il Kim* · Boram Yoon · Seo Young Oh · Woontack Woo†

IEEE Transactions on Visualization and Computer Graphics ( Volume: 29, Issue: 11, November 2023)Top 10%

In this paper, we present a prototype system for sharing a user's hand force in mixed reality (MR) remote collaboration on physical tasks, where hand force is estimated using wearable surface electromyography (sEMG) sensor. In a remote collaboration between a worker and an expert, hand activity plays a crucial role. However, the force exerted by the worker's hand has not been extensively investigated. Our sEMG-based system reliably captures the worker's hand force during physical tasks and conveys this information to the expert through hand force visualization, overlaid on the worker's view or on the worker's avatar. A user study was conducted to evaluate the impact of visualizing a worker's hand force on collaboration, employing three distinct visualization methods across two view modes. Our findings demonstrate that sensing and sharing hand force in MR remote collaboration improves the expert's awareness of the worker's task, significantly enhances the expert's perception of the collaborator's hand force and the weight of the interacting object, and promotes a heightened sense of social presence for the expert. Based on the findings, we provide design implications for future mixed reality remote collaboration systems that incorporate hand force sensing and visualization.

가상 박물관에서 예술 작품의 콘텐츠 유사성에 기반한 공간 적응형 자동 배치 방법 및 그 시스템Pat.
우운택 · 김하연 · 마리암 샤케리

Application No. 10-2023-0100227 · Korea · Pending

본 발명은 예술 작품의 유사도와 3차원 가상 박물관 환경의 공간적 특성을 고려하여 전시의 테마 공간에 따라 예술 작품을 자동 배치하는 가상 박물관에서 예술 작품의 콘텐츠 유사성에 기반한 공간 적응형 자동 배치 방법 및 그 시스템에 관한 것으로서, 가상 박물관에 배치할 예술 작품의 예술품 데이터를 수신하는 단계, 상기 예술품 데이터에 기반하여, 예술품 간의 유사도를 산출하는 단계, 기 설정된 클러스터 크기에 따른 제약 조건으로 상기 예술품 데이터를 클러스터링하는 단계 및 클러스터링된 예술품 클러스터와 사전 정의된 공간 데이터를 이용한 최적화 접근 방식으로 3차원 가상 박물관 환경에 예술품을 배치하여 테마 공간 레이아웃을 구성하는 단계를 포함한다.

노안자와 난청자를 위한 시선 추적 및 음향 여과 기술을 이용한 AI 확장현실(XR) 시스템Pat.
우운택 · 신승엽

Application No. 10-2023-0162977 · Korea · Pending

모바일 디바이스를 이용한 증강 현실 기반 가이드 방법Pat.
우운택 · 김재인

KR 10-2583243-0000 · Korea · Granted

모바일 디바이스를 이용한 증강 현실 (Augmented reality, AR) 기반 가이드 방법이 제공된다. 상기 방법은, 모바일 디바이스가, 관심 객체 (Point of Interest, PoI) 로부터 소정 반경 이내의 영역을 나타내는 관심 영역 (Region of Interest, RoI) 내에 위치하는지 여부를 결정한다. 모바일 디바이스가 관심 영역 내에 위치하지 않는다는 결정에 응답하여, 모바일 디바이스의 표시부에 관심 영역 및 관심 객체 중 적어도 하나와 모바일 디바이스의 현재 위치를 2 차원 지도상에서 표시하는 2 차원 지도 뷰 표시 모드를 수행하고, 모바일 디바이스가 관심 영역 내에 위치한다는 결정에 응답하여, 모바일 디바이스의 표시부에 모바일 디바이스의 카메라 뷰와 관심 객체 메뉴를 표시하는 증강 현실 뷰 표시 모드를 수행할 수 있다.

모션 블러된 RGB 이미지에서의 3D 손 메시 복원 방법 및 장치Pat.
우운택 · 박갑용 · 김태균

KR 10-2514807-0000 · Korea · Granted

모션 블러된 RGB 이미지에서의 3D 손 메시 복원 방법 및 장치가 제시된다. 일 실시예에 따른 컴퓨터 시스템으로 구현되는 3D 손 메시 복원 장치에서 수행되는 모션 블러된 RGB 이미지에서의 3D 손 메시 복원 방법은, 모션 블러가 발생하는 RGB 이미지에서 선명한 이미지 및 흐릿한 이미지 쌍의 데이터셋을 생성하는 단계; 및 생성된 상기 데이터셋을 이용하여 디블러링(deblurring) 모듈 및 손 메시 복원(hand mesh recovery) 모듈을 학습하는 단계를 포함하고, 상기 디블러링 모듈 및 손 메시 복원 모듈을 통해 상기 모션 블러가 발생하는 RGB 이미지에서 손 자세 또는 메시를 추정할 수 있다.

사실적 혼합현실 라이팅을 위한 단일 재질 객체로부터의 딥러닝 기반의 조명 조건 추정 방법 및 시스템Pat.
우운택 · 박진우

KR 10-2558171-0000 · Korea · Granted

본 발명은 물리 기반의 딥러닝 네트워크를 통해 재질 정보를 제거하여 대상 객체에서 반사되는 순수한 빛 정보로부터 보다 정확한 조명 조건을 추정하는 기술에 관한 것으로, 입력 이미지에 포함된 단일 재질의 대상 객체를 검출하며, 상기 대상 객체로부터 재질 정보를 제거하여 빛 반사 정보를 획득하는 단계, 상기 대상 객체에서 반사되는 상기 빛 반사 정보에서 파노라마 이미지 형태의 LDR(Low Dynamic Range) 빛 정보를 추정하는 단계 및 상기 LDR 빛 정보를 이용하여 HDR(High Dynamic Range) 빛 정보를 추정하며, 상기 LDR 빛 정보 및 상기 HDR 빛 정보를 합하여 상기 대상 객체에 대한 전방향 빛 정보를 추정하는 단계를 포함한다.

서로 다른 공간 등록을 위한 에지 중심 공간 재설정 방법 및 시스템Pat.
우운택 · 김두영

Application No. 10-2023-0100226 · Korea · Pending

본 발명은 방향 전환 보행으로 물리적 공간을 조정하는 효과를 활용하여 서로 다른 물리적 - 가상 공간을 등록하는 새로운 공간 재설정 기법인 에지 중심 공간 재설정 방법 및 시스템에 관한 것으로서, 객체의 기본 기하학적 정보를 기준으로 물리적 공간을 그리드 단위로 분할하는 단계, 리디렉션(ReDirected Walking; RDW)에서 분할된 물리적 공간 그리드에 상대적 변환 이득(Relative Translation Gains; RTGs)을 적용하는 단계 및 가상 공간과 물리적 공간 간 좌표계를 유지하면서 가상 공간의 관심 대상을 물리적 공간의 해당 대상에 등록하여 가상 공간을 상기 물리적 공간으로 재설정하는 단계를 포함한다.

스마트 안경에서의 증강현실 알림 배치 방법Pat.
우운택 · 이현진 · 방선영

Application No. 10-2023-0158412 · Korea · Pending

시각장애인을 위한 3D 재구성 및 객체 인식을 통한 확장현실(XR) 공간 설명 시스템Pat.
우운택 · 신승엽

Application No. 10-2023-0162941 · Korea · Pending

실시간 RGB-D 스트리밍을 이용한 3차원 벡터화 매핑 방법 및 시스템Pat.
우운택 · 신승엽

Application No. 10-2023-0160116 · Korea · Pending

본 발명은 XR 장비의 RGB-D 데이터 스트리밍과 vMAP 기술을 결합하여 실시간 3차원 환경 매핑을 가능하게 하는 실시간 RGB-D Streaming을 이용한 3차원 벡터화 매핑 방법 및 시스템에 관한 것으로서, XR 장비를 활용하여, vMAP 처리 유닛에 필요한 센서 데이터를 수집하는 단계 및 수집된 센서 데이터를 활용하여 vMAP 처리 유닛을 통하여 벡터화된 3차원 맵을 생성하는 단계를 포함한다.

의미론적 객체 매칭을 통한 파노라마 텍스처 매핑 방법 및 시스템Pat.
우운택 · 박진우

KR 10-2558095-0000 · Korea · Granted

본 발명은 의미론적 객체 매칭을 통한 파노라마 텍스처 매핑 방법 및 시스템에 관한 것으로, 거리 뷰 이미지에서 의미론적 정보를 추출하는 단계, 사용자의 움직임에 따른 카메라 뷰 이미지에 대한 G-버퍼를 렌더링하며, 상기 G-버퍼에서 각 픽셀을 상기 거리 뷰 이미지로 매핑하는 파노라마 이미지 매핑을 수행하는 단계 및 상기 의미론적 정보와 상기 G-버퍼를 이용하여 깊이 테스트, 의미 객체 매칭 테스트 및 3D 인페인팅(inpainting)을 통해 상기 카메라 뷰 이미지와 상기 거리 뷰 이미지에 대한 파노라마 텍스처 매핑을 수행하는 단계를 포함한다.

지리정보를 활용한 대화형 디지털트윈 방법 및 장치Pat.
우운택 · 신승엽

KR 10-2515233-0000 · Korea · Granted

지리정보를 활용한 대화형 디지털트윈 방법 및 장치가 제시된다. 일 실시예에 따른 컴퓨터 장치에 의해 수행되는 지리정보를 활용한 대화형 디지털트윈 방법은, 2D 또는 3D로 변환된 지리정보 데이터를 획득하는 단계; 상기 지리정보 데이터를 단말에 시각화하여, 사용자에게 시각화된 정보를 제공하는 단계; 및 대화형 인터페이스를 이용한 인공지능 또는 음성인식 기반 챗봇을 통해 상기 사용자의 상기 단말과 상호 작용하여, 상기 시각화된 정보를 조작하거나 외부 시스템을 제어하는 단계를 포함하여 이루어질 수 있다.

확장 현실 다중 학습자 모니터링 시스템Pat.
우운택 · 김두영

Application No. 10-2023-0027986 · Korea · Pending

Seoyoung Kang* · Sungwoo Jeon · Woontack Woo†

IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW) 2023

We propose a Speech-Pose integration method to overcome the limitation of existing body pose estimation. Unlike previous Speech-based gesture generation method, our proposal reflects the user's actual pose using a vision-based system and speech as a subsidiary. When the system detects the camera out of sight, we carry out a context-aware method analyzing the speech. The system determines the target pose and connects it with the last pose captured by the camera using the bounding box. Our system can be a robust solution for avatar-mediated remote collaboration which requires accurate gesture delivery such as VR remote yoga training.

Hayun Kim* · Maryam Shakeri · Jaeeun Shin · Woontack Woo†

ACM Journal on Computing and Cultural Heritage

Virtual Reality (VR) provides curators with the tools to design immersive 3D exhibition spaces. However, manually positioning artworks in VR is labor-intensive, and most existing automated methods are limited in considering both artwork content and spatial characteristics, as well as accommodating curators’ design preferences. To address these challenges, we present a virtual exhibition authoring system that automatically generates optimized artwork placements for thematic space layouts, where exhibits are grouped by conceptual themes and arranged as clusters. Our approach clusters artworks based on five content similarity factors: color, material, description, artist, and production date, and allows curators to adjust the importance of each factor and set limits on cluster sizes according to their design goals. A genetic optimization algorithm is employed to determine the placement of artworks, using four cost functions: intra-cluster distance, inter-cluster distance, intra-cluster intervisibility, and occupancy, to evaluate the arrangement with respect to spatial characteristics. The effectiveness of our approach is demonstrated through a series of practical scenarios and an expert evaluation with curators.

202220

Dooyoung Kim* · Seonji Kim · Jaeeun Shin · Boram Yoon · Jinwook Kim · Jeongmi Lee · Woontack Woo†

Springer Virtual Reality

In this study, we explore how spatial configurations can be reflected in determining the threshold range of Relative Translation Gains (RTGs), a translation gain-based Redirected Walking (RDW) technique that scales the user’s movement in Virtual Reality (VR) in different ratios for width and depth. While previous works have shown that various cognitive factors influence the RDW threshold, studies investigating the impact of environmental aspects on the RDW threshold with regard to the user’s visual perception were lacking. Therefore, we examined the effect of spatial configurations on the RTG threshold by analyzing the participant’s responses and gaze distribution data in two user studies. The first study concerned the size of the virtual room and the existence of objects within it, and the second study focused on the combined impact of room size and spatial layout. Our results show that room size, the existence of objects, and spatial layout all significantly affect the RTG threshold range. Based on our findings, we propose virtual space rescaling guidelines to increase the range of adjustable movable space with RTGs for developers: placing distractors in the room, setting the perceived movable space to be larger than the adjusted movable space for an empty room, and avoiding the placement of object clusters in the center of the room. Our findings can be used to adaptively rescale VR users’ space according to the target virtual space’s configuration with a unified coordinate system that enables the utilization of physical objects in a virtual scene.

Hyunjin Lee* · Woontack Woo†

IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct) 2022

Augmented reality head-mounted displays (AR HMD) enable visual notifications directly into the user's field of view (FoV) to perform dual-task. However, research on the efficient presentation of notifications when communicating with a virtual avatar in augmented reality (AR) situation has been insufficient. For efficient AR notification presentations, it is necessary to understand where to display notifications without interfering with virtual avatar communication. We investigated two coordinate systems (world-fixed and display-fixed) and eight positions for AR notifications when conversing with the virtual avatar. Results showed that the top-left position increased response time and task load. Our study contributes as a cornerstone to designing AR notifications in communication with virtual avatars.

Dooyoung Kim* · Hyung-il Kim · Woontack Woo†

IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct) 2022

This study proposed a Mixed Reality (MR) mutual space gener-ation method that aligns the Virtual Reality (VR) clients' spaces around the Augmented Reality (AR) host space's reference object and adjusts the virtual space's scale with translation-based Redi-rected Walking (RDW). Although there were methods to create shared spaces by finding optimal transform or each space to max-imize the semantically aligned area, there was a lack of research on utilizing the effect of rescaling a VR client's space with the RDW technique. We leveraged Relative Translation Gains (RTGs), a length-based RDW technique, to maximize the semantic information matching ratio between the AR host space and the VR clients' spaces by rescaling the client's virtual space within a suitable threshold range according to the spatial configuration. Our pilot study results show that the semantic information matching rate between the AR and the VR space increased compared to the existing one-to-one scale matching method while maintaining a unified coordinate sys-tem. Our findings can be used to create a more expansive mutual space for asymmetric remote collaboration even if remote spaces increase and challenging to secure a sufficient shared area.

Jaeeun Shin* · Boram Yoon · Dooyoung Kim · Hyung-il Kim · Woontack Woo†

IEEE International Symposium on Mixed and Augmented Reality (ISMAR) 2022Top-tier

This paper evaluates factors of social presence during a dynamic remote collaboration task in a technologically asymmetric Mixed Reality (MR) setting for two spatial layouts. While active movement during MR remote collaboration is afforded by how the shared 3D space is mediated and configured, studies investigating the impact of these conditions on user experience have been scarce. In a between-group study $(\mathrm{n}=48)$, a host user in Augmented Reality (AR) and a remote user in Virtual Reality (VR), both wearing Head Mounted Displays (HMDs), simultaneously moved around the shared space to find and assemble parts of a Mars exploration rover together, one group in a Peripheral layout and the other in a Scattered layout with disparate levels of spatial affordance. Results show that while VR facilitates higher co-presence and spatial presence than AR through HMDs, the Peripheral layout enables users to pay more attention to one another than the Scattered. We analyze the results and derive implications aimed at bridging the AR-VR gap in social presence for dynamic MR remote collaboration through the adaptive placement of virtual content in shared spaces.

Boram Yoon* · Seonji Kim · Woontack Woo†

IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct) 2022

We present a user interface method called TranSlider, designed for an AR collaboration with the concept of customized visualization of the user-selected augmented content. In AR, virtual content is augmented on top of the real world, so collaborating users are inevitably exposed to the diverse-overlapped information layers. Because the human recognition system has limited capacity, this overloaded information may burden the user to understand fully, and it is also challenging to manipulate among occluded objects. To solve this issue, we designed the TranSlider that allows users to control the presence of augmented content being transparent or highlighted based on the user interaction. The implemented prototype includes UI components, a manipulation function, and a network-connected collaboration module, and its applicability was verified through a preliminary interview. We expect that it would effectively support general AR collaborations and those with specific purposes, such as designing complex, hierarchical, or made up of many partial assets.

Seo Young Oh* · Boram Yoon · Woontack Woo†

IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct) 2022

We propose a system concept for Augmented Reality Head-Mounted Display users, which supports multitask viewing with multiple vir-tual workspaces anchored in the real-world space. Although people encounter multitasking necessities frequently, the native AR HMD and existing interfaces lack measures to visualize multiple sets of spatially-anchored information in parallel. The system separately vi-sualizes two different sets of spatially-anchored information, one on each AR HMD and smartphone, enabling side-by-side multitasking on AR HMD without applying heavy load on users. We implemented a proof-of-concept prototype that allows side-by-side viewing of the two different virtual workspaces. The proposed concept shows promises of multitasking on AR HMD, and future research will de-velop the system to be fully functional and verified with user studies.

Seonji Kim* · Hyuckjin Jang* · Kyung Taek Oh* · Seo Young Oh · Dooyoung Kim · Woontack Woo · Jeongmi Lee · Jaehong Ahn · Sang Ho Yoon†

IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct) 2022

We propose an AR store authoring system that enables the users to customize a physical space to promote spatial experience in mobile AR shopping. The current state of AR shopping research attempts to enhance the shopping experience by presenting virtual items onto the user's space. However, under the notion that the stores and products are inseparable in creating a holistic shopping experience, we set to make a shopping experience that encompasses both elements. Instead of simply augmenting a whole store, this study introduces a space adaptive AR store that divides a virtual store into multiple sections and places them onto physical space according to the labeled surfaces. This prototype mobile application intends to deliver a complete spatial shopping experience by allowing the users to customize the stores in their space and ultimately attach the store experience to the AR shopping system. This study aims to introduce new scalability of the AR technology application in remote shopping.

Seoyoung Kang* · Emmanuel Ian Libao · Juyoung Lee · Woontack Woo†

IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct) 2022

We propose DirectionQ, an interaction technique using a continuous mid-air hand input that shows directional visual cues on objects to enable fast and accurate multiple target selection in dense environments. The user can select various targets in a row by moving their hand consecutively in the same direction with the visual cues of targeted objects. DirectionQ involves assigning directional visual cues to each object in the user's viewpoint, detecting the trajectory changes of hand movement, and inferring intended targets with directions of the segmented hand movement. Unlike traditional approaches for multiple target selection, DirectionQ does not need additional steps to disambiguate, providing less time for multiple target selection. While distant, small, or occluded objects often bring accuracy or time-consuming problems, DirectionQ shows similar performance regardless of target conditions. To prove the effectiveness of our approach, we conducted two controlled experiments comparing our system with two existing techniques, RayCasting and EyeGaze. The results show that DirectionQ is faster and more accurate than two techniques with a lower cognitive load on multiple target selection.

Seoyoung Kang* · Boram Yoon · Bowon Kim · Woontack Woo†

IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct) 2022

This study explored how avatar face Level of Detail (LOD) and task types affect users' social presence and interpersonal communication in Augmented Reality remote collaboration. In AR remote collaboration, users are represented as avatars, and an actual user's appearance inevitably changes. While certain situations require an avatar that resembles a user, it is challenging to reconstruct everything realistically due to its high cost. Since facial parts show relative significance compared to body parts, we proposed controlling face LOD as an approach to lowering the cost while maintaining social presence in AR environments. Our results showed that while Mid-LOD maintains social presence but reduces costs compared to High-LOD, Low-LOD negatively impacts social presence mainly due to emotional understanding. Based on our findings, we highlight the importance of 1) considering facial expressions and emotional understanding in managing face LOD, 2) figuring out the collaboration context with different effects of tasks and communication behaviors, and 3) providing additional communication cues for maintaining social presence while lowering the cost.

Hayun Kim* · Gun Lee · Mark Billinghurst · Woontack Woo†

IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct) 2022

Recent advances in digital heritage have led to participatory interpretation that enables the creation of content provided by individuals as well as content from official cultural institutions. Although researchers have developed systems supporting participatory interpretation, little is known about how people experience participatory interpretation that is socially generated. To investigate this, we developed a mobile Virtual Reality application called MyInterpret and conducted a user study with 17 participants. The study results showed that user-generated interpretation significantly improving user experience, participants created and perceived user-generated interpretation differently from the official interpretation, and social features were used for engaging with user-generated content. We outline the implications of our work for curators and designers, and discuss how they may benefit from considering ‘participatory interpretation’ in the future.

Hyunjin Lee* · Sunyoung Bang · Woontack Woo†

Springer Virtual Reality

Augmented reality (AR) head-mounted displays (HMDs) allow users to easily receive notifications while participating other tasks by projecting information directly in their field of view. Although HMDs offer such benefits in displaying notifications, there is insufficient research on the effective placement of AR notifications when the user is walking. For this, we conducted two studies based on different types of AR information to identify how the users perceive and understand the AR notifications according to placement while walking. We compared two different coordinate systems (display-fixed and body-fixed) and three different positions (top, right, and bottom) for icon-type and text-type notifications. The results indicated that using a display-fixed coordinate system for icon-type notifications yields significantly higher noticeability and comprehension. In contrast, using a body-fixed coordinate system for text-type notifications significantly improved comprehension and walking performance. Regarding the position of notifications, the bottom position resulted in a significantly higher noticeability and comprehension for both icon- and text-type notifications compared with the top. Based on these results, we draw some recommendations for the future design of notifications in AR HMDs.

Dooyoung Kim* · Jinwook Kim · Jaeeun Shin · Boram Yoon · Jeongmi Lee · Woontack Woo†

IEEE Conference on Virtual Reality and 3D User Interfaces (VR) 2022Top-tier

This paper investigates how the size of virtual space and objects within it affect the threshold range of relative translation gains, a Redirected Walking (RDW) technique that scales the user’s movement in virtual space in different ratios for the width and depth. While previous studies assert that a virtual room’s size affects relative translation gain thresholds on account of the virtual horizon’s location, additional research is needed to explore this assumption through a structured approach to visual perception in Virtual Reality (VR). We estimate the relative translation gain thresholds in six spatial conditions configured by three room sizes and the presence of virtual objects (3 × 2), which were set according to differing Angles of Declination (AoDs) between eye-gaze and the forward-gaze. Results show that both size and virtual objects significantly affect the threshold range, it being greater in the large-sized condition and furnished condition. This indicates that the effect of relative translation gains can be further increased by constructing a perceived virtual movable space that is even larger than the adjusted virtual movable space and placing objects in it. Our study can be applied to adjust virtual spaces in synchronizing heterogeneous spaces without coordinate distortion where real and virtual objects can be leveraged to create realistic mutual spaces.

Hyunjin Lee* · Woontack Woo†

IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW) 2022

Providing additional information in the proper position of augmented reality (AR) head-mounted display (HMD) can help increase AR performance and usability for dual-task. Therefore, our study inves-tigated how to place notifications for the dual-task to address this. We compared eight display positions and two tasks (single and dual tasks) to identify the appropriate area for displaying notifications. We confirmed that the middle-right reduces response time and task load. In contrast, the top-left is the location, which should avoid providing any notification in AR dual-task. Our study contributes to designing AR notifications on HMDs to enhance everyday AR experiences.

Hyerim Park* · Maryam Shakeri · Ikbeom Jeon · Jangyoon Kim · Abolghasem Sadeghi-Niaraki · Woontack Woo†

Springer Virtual Reality

There have been attempts to provide new cinematic experiences by connecting TV or movie content to suitable locations through augmented reality (AR). However, few studies have suggested a method to manage breakdowns in continuity due to spatial transitions. Thus, we propose a method to manage the spatial transition that occurs when we create a TV show trajectory by mapping TV show scenes with spatiotemporal information to the real world. Our approach involved two steps. The first step is to reduce the spatial transition considering the sequence, location, and importance of TV show scenes when creating the TV show trajectory in the authoring tool. The second is to fill the spatial transition with additional TV show scenes considering sequence, importance, and user interest when providing the TV show trajectory in the mobile application. The user study results showed that reducing spatial transition increases narrative engagement by allowing participants to see important content within the trajectory. The additional content in spatial transition decreased the physical demand and effort in terms of the perceived workload, although it increased the task completion time. Integrated spatial transition management improved the overall cinematic augmented reality (CAR) experience of the TV show. Furthermore, we suggest design implications for realizing the CAR of TV shows based on our findings.

CONVERSATIONAL DIGITAL TWIN METHOD USING GEOGRAPHIC INFORMATION AND APPARATUS THEREOFPat.
우운택 · 신승엽

Other · Pending

Disclosed are a conversational digital twin method and apparatus using geographic information. A conversational digital twin method using geographic information, which is performed by a computer device, includes obtaining geographic information data converted into 2D or 3D; visualizing the geographic information data in a terminal and providing visualized information to a user; and interacting with the terminal of the user through an artificial intelligence or a voice recognition-based chatbot using a conversational interface to manipulate the visualized information or control an external system.

METHOD AND SYSTEM FOR SYNTHESIZING NOVEL VIEW IMAGE ON BASIS OF MULTIPLE 360 IMAGES FOR 6-DEGREES OF FREEDOM VIRTUAL REALITYPat.
Woontack Woo · Hochul Cho

Other · Pending

A method and a system for synthesizing a novel-view image based on multiple 360 images for 6DOF virtual reality, in which a large-scale 6-DOF virtual environment is implemented, and a scene is synthesized at a novel viewpoint, are provided. The method includes performing a 3D reconfiguration procedure for the 360 images to recover 3D geometric information, and to reconfigure a virtual data map in which the multiple 360 images are integrated into one image, producing a view image corresponding to a viewpoint of a user by applying a view synthesis algorithm of projection & vertex warping process using a reference image which is closest to a viewpoint extracted from the virtual data map, and bl

Jaeeun Shin* · Boram Yoon · Dooyoung Kim · Woontack Woo†

IEEE Transactions on Visualization and Computer GraphicsTop 10%

A critical yet unresolved challenge in designing space-adaptive narratives for Augmented Reality (AR) is to provide consistently immersive user experiences anywhere, regardless of physical features specific to a space. For this, we present a comprehensive analysis on a series of user studies investigating how the size, density, and layout of real indoor spaces affect users playing Fragments, a space-adaptive AR detective game. Based on the studies, we assert that moderate levels of traversability and visual complexity afforded in counteracting combinations of size and complexity are beneficial for narrative experience. To confirm our argument, we combined the experimental data of the studies (n=112) to compare how five different spatial complexity conditions impact narrative experience when applied to contrasting room sizes. Results show that whereas factors of narrative experience are rated significantly higher in relatively simple settings for a small space, they are less affected by complexity in a large space. Ultimately, we establish guidelines on the design and placement of space-adaptive augmentations in location-independent AR narratives to compensate for the lack or excess of affordances in various real spaces and enhance user experiences therein.

물체와 상호작용하는 손의 추적을 위한 맨손 깊이 인페인팅 방법 및 시스템Pat.
우운택 · 조우진 · 박갑용

KR 10-2395123-0000 · Korea · Granted

본 발명은 조건부 적대적 생성망(cGAN)으로 구현되는 맨손 깊이 인페인팅을 사용하여 물체와 상호작용하는 손을 3D 추적하는 손의 추적을 위한 맨손 깊이 인페인팅 방법 및 시스템에 관한 것으로, 물체와 상호작용하는 손에 대한 카메라 이미지를 수신하는 단계, 상기 카메라 이미지에서 손 및 물체를 추적하여 깊이 인페인팅(depth inpainting)을 이용한 맨손 깊이 이미지를 생성하는 단계 및 상기 맨손 깊이 이미지에서 손 포즈를 추정하는 단계를 포함한다.

실제 및 가상 사물인터넷 기기를 연동한 3차원 디지털 트윈 시각화 시스템 및 그 방법Pat.
우운택 · 신승엽 · 조호철

KR 10-2441454-0000 · Korea · Granted

본 발명은 실제 및 가상 사물인터넷 기기를 연동한 3차원 디지털 트윈 시각화 시스템 및 그 방법에 관한 것으로서, 실세계에 배치되는 실제 사물인터넷(IoT) 기기로부터 실제 지리공간 데이터를 수집하며, 가상세계에서 서버의 셀프 루틴을 통해 생성되는 가상 사물인터넷(IoT) 기기로부터 가상 지리공간 데이터를 수집하는 가상 및 실제 데이터 수집부, 상기 실제 지리공간 데이터 및 상기 가상 지리공간 데이터를 통합하여 도시 디지털 트윈(Urban Digital Twin; UDT)에 전송하는 데이터 처리부 및 상기 통합된 데이터를 기반으로 상호작용을 위한 증강현실의 동시 공간을 제공하며, 테이블탑(Tabletop)으로부터 제어를 위해 생성된 네트워크 패킷을 중계하는 도시 디지털 트윈부를 포함한다.

Yun Suen Pai* · Marsel L. Bait · Juyoung Lee · Jingjing Xu · Roshan L Peiris · Woontack Woo · Mark Billinghurst · Kai Kunze†

Springer Virtual Reality

We present NapWell, a Sleep Assistant using virtual reality (VR) to decrease sleep onset latency by providing a realistic imagery distraction prior to sleep onset. Our proposed prototype was built using commercial hardware and with relatively low cost, making it replicable for future works as well as paving the way for more low cost EOG-VR devices for sleep assistance. We conducted a user study (n = 20) by comparing different sleep conditions; no devices, sleeping mask, VR environment of the study room and preferred VR environment by the participant. During this period, we recorded the electrooculography (EOG) signal and sleep onset time using a finger tapping task (FTT). We found that VR was able to significantly decrease sleep onset latency. We also developed a machine learning model based on EOG signals that can predict sleep onset with a cross-validated accuracy of 70.03%. The presented study demonstrates the feasibility of VR to be used as a tool to decrease sleep onset latency, as well as the use of embedded EOG sensors with VR for automatic sleep detection.

202119

Hyung-il Kim* · Taehei Kim · Eunhwa Song · Seo Young Oh · Dooyoung Kim · Woontack Woo†

IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct) 2021

In this poster, we present a digital twin-based mixed reality system for remote collaboration with the size-scaling of the user and the space. The proposed system supports collaboration between an AR host user and a VR remote user by sharing a 3D digital twin of the AR host user. To enhance the coarse authoring of a shared digital twin environment, we provide a size scaling of the digital twin environment with the world-in-miniature view. Also, we enable scaling the size of the VR user’s avatar to enhance both coarse (size-up) and fine-grained (size-down) authoring of the digital twin environment. We describe the system setup, input methods, and interaction methods for scaling space and user.

Gi-Bbeum Lee* · Hyuckjin Jang · Hyundeok Jeong · Woontack Woo†

IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct) 2021

In remote collaboration using Augmented Reality (AR), speech and gesture are major communication methods for the general public. However, the Deaf and Hard-of-Hearing (DHH) population cannot join in the communication due to the absence of a sign language interface which is their primary language. Recent works have tried to augment spoken language with sign language animations or captions, but the research to convey sign language with spoken language is still very limited. In this paper, we propose a novel multi-modal communication system that integrates sign language translation, speech recognition, and shared object manipulation in the mobile AR environment. Though the system is currently under development, we demonstrated a rapid prototype of the telemedicine app leveraging the video prototyping method to integrate the system modules. We performed preliminary interviews about our approach with DHH users, a sign language interpreter, and a physician. We discuss the insights into the future design of the DHH communication support in the AR collaboration system. This study has a socio-cultural, economic impact on the DHH population as a barrier-free design of a remote collaboration system in a practical scenario. Another contribution of this work is that we suggested a novel user-centered system for DHH users in AR by integrating the existing technologies.

Seungyoub Ssin* · Hochul Cho · Woontack Woo†

Interaction Design and Architecture(s)

Despite prior efforts to solve urban problems and build smart cities using large-scale urban information, researchers have yet to develop tools for city administrators to easily access a diverse mix of city information. To address this limitation, this article proposes an augmented reality visualization system, GeoACT, which can develop a digital twin to present city information from both macro and micro perspectives. GeoACT employs a 3D virtual city miniature with real and virtual IoT information to provide a macro perspective view, and a 360-degree-video-based visualization technique is used for a micro perspective view. We applied GeoACT to an urban space and demonstrated that it could effectively visualize city information through an augmented reality Head Mounted Display (HMD). Furthermore, GeoACT can be used in urban control centers.

Jaeeun Shin* · Boram Yoon · Dooyoung Kim · Woontack Woo†

Proceedings of the 2021 CHI Conference on Human Factors in Computing SystemsTop-tier

Space-adaptive algorithms aim to effectively align the virtual with the real to provide immersive user experiences for Augmented Reality(AR) content across various physical spaces. While such measures are reliant on real spatial features, efforts to understand those features from the user’s perspective and reflect them in designing adaptive augmented spaces have been lacking. For this, we compared factors of narrative experience in six spatial conditions during the gameplay of Fragments, a space-adaptive AR detective game. Configured by size and furniture layout, each condition afforded disparate degrees of traversability and visibility. Results show that whereas centered furniture clusters are suitable for higher presence in sufficiently large rooms, the same layout leads to lower narrative engagement. Based on our findings, we suggest guidelines that can enhance the effects of space adaptivity by considering how users perceive and navigate augmented space generated from different physical environments.

Hui-Shyong Yeo* · Juyoung Lee · Woontack Woo · Hideki Koike · Aaron J Quigley · Kai Kunze†

Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems

In this work, we explore a new sensing technique for smart eyewear equipped with Electrooculography (EOG) sensors. We repurpose the EOG sensors embedded in a JINS MEME smart eyewear, originally designed to detect eye movement, to detect midair hand gestures. We also explore the potential of sensing human proximity, rubbing action and to differentiate materials and objects using this sensor. This new found sensing capabilities enable a various types of novel input and interaction scenarios for such wearable eyewear device, whether it is worn on body or resting on a desk.

Seungyoub Ssin* · Hochul Cho · Woontack Woo†

Augmented Reality and Virtual Reality

This paper introduces a method for developing Augmented Urban Digital Twin (A-UDT) for virtual and real Internet-of-Things (IoT) visualization and its benefits. It presents a method for generating virtual data using the Virtual IoT (VI) based on real IoT data, which can be used with cloud services. An urban digital twin is developed to find and simulate various problems arising in the city by extracting and visualizing data of the city. Designing urban digital twin has some challenging issues such as the difficulty of processing environmental information in real-time due to the absence of IoT support. This absence of IoT support can make problems like developing urban digital twin without knowing the types and scope of visualization. To overcome this limitation, we propose the augmented city miniature that uses generation system, process, storage, and network system of the component that generates VI with the same data structure as real IoT to generate and send data, and it simulates information on a 3D geographic map by sending the generated virtual data to the visualization part using Augmented Reality (AR). Urban digital twin based on the virtual data can make various simulations of city visualization that can be performed to discover required IoT and reduce development costs. A-UDT system can be applied to monitor and simulate diverse environmental information such as air quality, energy efficiency, object movements, and temperature distribution.

Seungyoub Ssin* · Minjeong Suh · Jongwook Lee · Timothy Jung · Woontack Woo†

Augmented Reality and Virtual Reality

The purpose of this study is to propose a theoretical framework for a digital twin-based smart science tourism system using augmented reality in Gwanghwamun Square. To date, smart tourism service was not sufficiently developed for the provision of relevant tourism information to visitors as tourism contents were managed by the separated information system of each institution. The tourists could not receive ideal tourism routes based on their preferences and also according to congestion of each institution. Moreover, the business model using existing tourism services was not well developed, and thus the current tourism services have not sufficiently contributed to the revitalization of the regional economy. We designed a digital twin-based tourism content management system that provides integrated management of tourism information, and augmented reality (AR) science tour services which are included with contents recommendation, best path finding, contents sharing and gamification. We also adopted a business model using the AR tourism platform based on digital twin technology which could contribute to the invigoration of the local economy.

Dooyoung Kim* · Jaeeun Shin · Jeongmi Lee · Woontack Woo†

IEEE Virtual Reality and 3D User Interfaces (VR) 2021

We propose the concept of relative translation gains, a novel Redirected Walking (RDW) method to create a mutual movable space between the Augmented Reality (AR) host's reference space and the Virtual Reality (VR) client's space. Previous RDW methods have focused on maximizing the movable space at the expense of aligning the coordinates between the AR and VR side, and could only be applied to collaborative scenarios involving sequential tasks. Our method solves these problems by adjusting the remote client's walking speed for each axis of a VR space to modify the movable area without coordinate distortion. We estimate the relative translation gain threshold, defined as the extent to which the walking speed can be altered without creating a perceived difference in distance. In order to reflect features of the reference space in generating the mutual space, we then examine how changing its size affects the threshold value. Our study showed that for remote clients connected to the larger reference space, relative translation gains can be increased to utilize a VR space bigger than their real space. Our method can be applied to create optimal mutual spaces for a wider variety of asymmetric Mixed Reality (MR) remote collaboration systems.

Hyerim Park* · Woontack Woo†

IEEE Virtual Reality and 3D User Interfaces (VR) 2021

Most research on providing location-based content in 3D interactive virtual reality has been limited to social media content. Few studies have suggested how to represent the video clip of movies or TV shows in virtual reality. This paper investigates a video content representation method to provide a hyper-reality experience of the narrative world in virtual reality. We reflect the time and place settings of the video content in virtual reality and have participants watch the video in four different virtual reality environments. We reveal that reflecting the story's environment settings to the virtual reality environment significantly improves the spatial presence and narratives engagement. We also confirm a positive correlation between spatial presence and narrative engagement, including subscales such as emotional engagement and narrative presence. Based on the study results, we discuss how to provide the hyper-reality experience in content-adaptive virtual reality.

Inhwa Yeom* · Woontack Woo†

IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW) 2021

We present a digital twin-based Mixed Reality platform for arts curators. Despite the rising presence of exhibitions in varying digital formats, the virtual engagement of arts curators has been nearly disabled. To replicate and enhance the professional capability of arts curators in the virtual realm, our system integrates a digital twin art space and 3D authoring techniques, aided with spatial and semantic metadata. Our user evaluation proves the system's usability, and its capacity to support curatorial activities. With this, we aim to provide a groundwork to similar systems that also extend arts curator's creativity and outreach beyond time and space.

Seungyoub Ssin* · Hochul Cho · Woontack Woo†

IEEE International Conference on Consumer Electronics (ICCE) 2021

This paper shows a structure and implementation of an Urban Digital Twin (UDT), simulating virtual data productions using Virtual IoT (VI) to design services for citizens. When designing smart city services with UDT systems, developing the system without data support from real IoTs can be created without knowing the scope and limitations of the IoTs. We simulated the services by creating VI system, which can be introduced in the future based on the meaningful data sets of existing Real IoT sensors. This paper focuses on the concept of the proposed system, GeoVCM, the components that make up the system.

HAND POSE ESTIMATION METHOD WITH A SINGLE INFRARED CAMERA VIA DOMAIN TRANSFER LEARNING AND THE SYSTEM THEREOFPat.
우운택 · 박갑용 · 김태균

PCT/KR2021/095031 · PCT · Pending

The present invention relates to a hand position estimation method and system for estimating a 3-dimensional hand position for quick hand movement, on the basis of domain transfer learning from a depth image to an infrared image, the hand position estimation method comprising the steps of: processing a depth image and an infrared image for hand movement; synthesizing a depth map with the infrared image by using a hand image generator (HIG), and estimating skeletal positions of hand joints from each of the depth map and an infrared map; and calculating a 3-dimensional hand position by using the skeletal positions and the center of a hand depth image.

Jinwoo Park* · Ikbeom Jeon · Sung-Eui Yoon · Woontack Woo†

IEEE Transactions on Visualization and Computer GraphicsTop 10%

증강현실 기반 인터렉티브 저작 서비스 제공 시스템Pat.
우운택 · 길경원 · 하태진 · 도영임 · 임지민

KR 10-2304023-0000 · Korea · Granted

본 발명은 HMD(Head Mounted Display)가 구비된 웨어러블(wearable) 디바이스와, 상기 웨어러블 디바이스와 페어링(pairing)되어 GUI 인터페이스를 통해 시나리오 기반 기설정된 플로우(flow)에 대응하는 콘텐츠를 재생하고, 상기 콘텐츠에 형성된 오브젝트에 인터럽트(interrupt) 발생 시 상기 웨어러블 디바이스로부터 뷰잉(viewing) 되는 3차원 공간에 해당 오브젝트를 오버레이(overlay)하여 증강현실 영상을 생성하고, 사용자 제스처(gesture)에 따라 오버레이된 오브젝트별 상태를 변환하고, 움직임 센서를 통해 센싱되는 움직임 정보를 기반으로 상기 오브젝트의 위치 영역을 변환하는 증강현실 서비스 제공 단말과, 자기 센서(magnetic sensor)를 구비하고, 상기 증강현실 서비스 제공 단말로부터 출력되는 오브젝트를 선택하거나 활성화하는 포인팅 디바이스를 포함함을 특징으로 한다.

증강현실을 이용한 기타 운지 학습 시스템Pat.
우운택 · 도승원 · 김형일

KR 10-2228019-0000 · Korea · Granted

본원의 일 측면에 따른 증강현실을 이용한 기타 운지 학습 시스템은 기타 영상을 촬영하는 카메라 모듈; 입력 모듈; 증강현실 영상이 출력되는 디스플레이 모듈; 증강현실을 이용한 기타 운지 학습 프로그램이 탑재된 메모리; 및 프로그램을 실행하는 프로세서를 포함한다. 기타는 복수의 스트링을 포함하고 지판의 일측에 증강현실 마커가 부착된 것이고, 프로세서는 기타 운지 학습 프로그램의 실행에 따라, 기타 영상에서 증강현실 마커를 인식하고, 증강현실 마커를 기초로 지판 상의 스트링 영역 및 프렛 영역을 추출하여 원본 이미지로 저장하고, 입력 모듈을 통해 입력 받은 악보 파일에서 곡의 박자, 운지 또는 주법이 포함된 연주 정보를 추출하고, 추출된 연주 정보를 이용하여 지판에서 운지해야 하는 위치 및 탄현해야 하는 스트링을 알려주는 표준 운지 이미지를 생성하고, 표준 운지 이미지와 원본 이미지를 중첩시킨 증강현실 영상을 생성하고, 생성된 증강현실 영상을 디스플레이 모듈을 통해 출력하는 것을 특징으로 한다.

한국HCI학회 학술대회 발표 논문집, 2021

Hyunju Kim* · Sanghwa Hong · Junki Kim · Taesoo Jang · Woontack Woo · Seongkook Heo · Byungjoo Lee†

Multimedia Tools and Applications

This study introduces RealityBrush, a novel augmented reality (AR) authoring system that allows designers to quickly and easily create realistic virtual objects by capturing and utilizing the kinetic properties of everyday physical objects in the early stages of design. The RealityBrush system consists of a handheld device, a data analysis module and an AR feedback module. The handheld device, which is made in the shape of a rod, is equipped with a depth camera and a force sensor at the tip. When a user holds the device and pokes a physical object, the local force applied to the object and the resulting deformations of the object are measured simultaneously. By analyzing the relationship between measured force and deformations, the RealityBrush system can identify two kinetic properties of the poked object: stiffness and motion resistance. The user can then use the handheld device as a 3D brush to create a virtual object in the air and assign the measured kinetic properties to the created virtual object. Finally, the system’s physics engine allows the user to interact with the created object by using the device to poke or push the object. The technical evaluation showed that the system can successfully extract the stiffness and motion resistance of everyday objects. We also report initial user feedback on AR authoring using the RealityBrush system.

202015

Gabyong Park* · Tae-Kyun Kim · Woontack Woo†

IEEE International Symposium on Mixed and Augmented Reality (ISMAR) 2020Top-tier

Previous methods successfully estimated 3D hand poses from unblurred depth images with slow and smooth hand motions. However, the performance drops when the depth images are contaminated by motion blur due to fast hand motion. In this paper, we exploit an infrared (IR) image input, which is weakly blurred under fast hand motion. The proposed method is based on domain transfer learning from depth to infrared images. Note we do not have IR images with hand skeletons, thus proposing self-supervision rather than direct supervision using the skeleton labels. We train a Hand Image Generator (HIG) and two Hand Pose Estimators (HPEs) on paired depth and infrared images via self-supervision using a consistency loss, guided by an existing HPE trained on paired depth and hand skeleton entries. The IR-based HPE is then refined on the weakly blurred infrared images. The qualitative and quantitative experiments demonstrate that the proposed method accurately estimates 3D hand poses under motion blur by fast hand motion, while existing depth-based methods fail. Our solution therefore supports fast 3D manipulation of virtual objects for augmented reality applications. Our model and dataset are publicly available for future research.1

Woojin Cho* · Gabyong Park · Woontack Woo†

IEEE International Symposium on Mixed and Augmented Reality (ISMAR) 2020Top-tier

We propose a 3D hand tracking system using bare-hand depth inpainting from an RGB-depth image for a hand interacting with an object. The effectiveness of most existing hand-object tracking methods is impeded by the insufficiency of data, which do not include hand data occluded by the object, and their reliance on the information inferred from assuming the specific object type. We generate a sufficiently accurate bare-hand depth image from a hand interacting with an object using a conditional generative adversarial network, which is trained using the synthesized 2D silhouettes of the object to learn the morphology of the hand. We evaluate the proposed approach using a hierarchical particle filter-based hand tracker and prove that our approach utilizing the bare-hand tracker in the hand-object interaction dataset achieve state-of-the-art performance. The generalization of our work will enable visual-tactile interaction that is more natural in various wearable augmented reality applications.

Sunyoung Bang* · Hyunjin Lee · Woontack Woo†

IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct) 2020

Using an augmented reality head-mounted display(HMD) to extend the display of smartphone could open up new user interface solutions, benefiting users with an increased screen real-estate and visualizations that leverage each device's capabilities. Some previous works have explored the viability of extended displays, but knowledge regarding considerable design factors and constraints for such displays is still very limited. In our work, we conducted an exploratory study to investigate how different properties of augmented content in extended displays affect the user's task performance and subjective workload in an image comparison task. Through an experiment with 24 participants, we compared four augmented content placements (top, right, left, and bottom) and two augmented content sizes (small and large). The study results demonstrated that there are significant effects of both placement and size of augmented content on task performance and subjective workload. Based on the findings of our study, we propose some recommendations for the future design of extended displays.

Hyunjin Lee* · Sunyoung Bang · Woontack Woo†

IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct) 2020

Information in augmented reality (AR) consists of virtual and real contexts and is provided to the AR environment at different distances for each user. Therefore, it is important to understand how the integration of two different information influences the user's AR experience. However, little has been studied regarding the complexity and viewing distance of a real space. Our study investigates how the complexity of the physical environment and viewing distance influence workload and performance in a visual search task in an AR environment. We conducted an experiment in which participants performed conjunction search under three different levels of background complexity, at both near (1.5m) and far (3m) distances. The results indicated that as the complexity of the background increased, the users' performance time and workload were negatively impacted. In addition, when the distance between the user and the background was greater, search time increased. From the results of the study, we derived some recommendations for the design of AR interfaces. Our research contributes to the design of interfaces by demonstrating the necessity to consider the complexity of background and viewing distance.

Boram Yoon* · Hyung-il Kim · Seo Young Oh · Woontack Woo†

IEEE International Symposium on Mixed and Augmented Reality (ISMAR) 2020Top-tier

This study investigates the effects of a virtual hand representation on the user experience including social presence during hand-based 3D remote collaboration. Although a remote hand appearance is a critical parts of a hand-based telepresence, it has been rarely studied in comparison to studies on the self-embodiment of virtual hands in a 3D environment. Thus, we conducted a user study comparing the three virtual hands models (Skeleton, Low Polygon and Realistic) while performing a remote collaborative task based on the American Sign Language (ASL) using both Augmented Reality (AR) and Virtual Reality (VR) environments. We found that the realistic type was perceived as the most sense of being together, human-like, and trustable representation. The low polygon model could also convey a clear sign and moderate level of social presence. Although the system was configured asymmetrically in AR and VR, little difference in perception was found except for the participant's mental load and message understanding. We then discuss the results and suggest design implications for future hand-based 3D telepresence systems.

Seo Young Oh* · Boram Yoon · Hyung-il Kim · Woontack Woo†

Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems

This paper reports that the time-domain accuracy of bare-hand interactions in HMD-based Augmented Reality can be improved by using finger contact: touching a finger with another or tapping one's own hand. The activation of input can be precisely defined by the moment of finger contact, allowing the user to perform the input precisely at the desired moment. Finger contact is better suited to the user's mental model, and natural tactile feedback from the fingertip also benefits the user with the self-perception of the input. The experimental results revealed that using finger contact is the preferred method of input that increases the time-domain accuracy and enables the user to be aware of the moment the input is activated.

Richard Li* · Juyoung Lee · Woontack Woo · Thad Starner†

Proceedings of the Augmented Humans International Conference

Cheek kissing is a common greeting in many countries around the world. Many parameters are involved when performing the kiss, such as which side to begin the kiss on and how many times the kiss is performed. These parameters can be used to infer one's social and physical context. In this paper, we present KissGlass, a system that leverages off-the-shelf smart glasses to recognize different kinds of cheek kissing gestures. Using a dataset we collected with 5 participants performing 10 gestures, our system obtains 83.0% accuracy in 10-fold cross validation and 74.33% accuracy in a leave-one-user-out user independent evaluation.

Hui-Shyong Yeo* · Juyoung Lee · Andrea Bianchi · Alejandro Samboy · Hideki Koike · Woontack Woo · Aaron Quigley†

Proceedings of the Augmented Humans International Conference

WristLens is a system for surface interaction from wrist-worn wearable devices such as smartwatches and fitness trackers. It enables eyes-free, single-handed gestures on surfaces, using an optical motion sensor embedded in a wrist-strap. This allows the user to leverage any proximate surface, including their own body, for input and interaction. An experimental study was conducted to measure the performance of gesture interaction on three different body parts. Our results show that directional gestures are accurately recognized but less so for shape gestures. Finally, we explore the interaction design space enabled by WristLens, and demonstrate novel use cases and applications, such as on-body interaction, bimanual interaction, cursor control and 3D measurement.

360 카메라 및 디지털 트윈을 기반으로 한 이동 객체의 지리적 데이터 추정방법 및 시스템Pat.
우운택 · 조호철 · 신승엽

Application No. 10-2020-0177977 · Korea · Pending

본 발명은 360 카메라 및 디지털 트윈을 기반으로 한 이동 객체의 지리적 데이터 추정방법 및 시스템에 관한 것으로, 360 카메라에서 영상 데이터를 수신하여 전처리하는 단계, 상기 영상 데이터 내 객체의 위치를 추출하며, 디지털 트윈(Digital Twin)을 통해 이미지 깊이(depth)를 추정하여 상기 객체의 초기 위치를 추정하는 단계 및 상기 초기 위치를 기반으로 상기 디지털 트윈에 배치된 3D 모델(3D model)을 이용하여 상기 영상 데이터 내 상기 객체와의 차이를 계산하며, 차이를 최소화하는 최적화(optimization)를 통해 상기 객체의 위치 및 각도를 포함하는 지리적 데이터를 추정하는 단계를 포함한다.

Gabyong Park* · Antonis Argyros · Juyoung Lee · Woontack Woo†

IEEE Transactions on Visualization and Computer GraphicsTop 10%

We present a sensor-fusion method that exploits a depth camera and a gyroscope to track the articulation of a hand in the presence of excessive motion blur. In case of slow and smooth hand motions, the existing methods estimate the hand pose fairly accurately and robustly, despite challenges due to the high dimensionality of the problem, self-occlusions, uniform appearance of hand parts, etc. However, the accuracy of hand pose estimation drops considerably for fast-moving hands because the depth image is severely distorted due to motion blur. Moreover, when hands move fast, the actual hand pose is far from the one estimated in the previous frame, therefore the assumption of temporal continuity on which tracking methods rely, is not valid. In this paper, we track fast-moving hands with the combination of a gyroscope and a depth camera. As a first step, we calibrate a depth camera and a gyroscope attached to a hand so as to identify their time and pose offsets. Following that, we fuse the rotation information of the calibrated gyroscope with model-based hierarchical particle filter tracking. A series of quantitative and qualitative experiments demonstrate that the proposed method performs more accurately and robustly in the presence of motion blur, when compared to state of the art algorithms, especially in the case of very fast hand rotations.

NOVEL VIEW SYNTHESIS METHOD BASED ON MULTIPLE 360 IMAGES FOR 6-DOF VIRTUAL REALITY AND THE SYSTEM THEREOFPat.
우운택 · 조호철 · 김장윤

PCT/KR2020/002289 · PCT · Pending

The present invention relates to a method and a system for synthesizing a novel view image on the basis of multiple 360 images for 6-degrees of freedom (DoF) virtual reality, wherein multiple 360 images are used to construct a large-scale 6-DoF virtual environment and synthesize a scene at a new viewpoint. The method comprises the steps of: performing a 3D reconfiguration procedure on 360 images to reconstruct 3D geometric information and reconfigure a virtual data map integrating the multiple 360 images into one image; using a reference image closest to a viewpoint extracted from the virtual data map to apply a projection 0026# vertex warping view synthesis algorithm so as to create view im

Jinwoo Park* · Hunmin Park · Sung-Eui Yoon · Woontack Woo†

IEEE Transactions on Visualization and Computer GraphicsTop 10%

In mixed reality (MR), augmenting virtual objects consistently with real-world illumination is one of the key factors that provide a realistic and immersive user experience. For this purpose, we propose a novel deep learning-based method to estimate high dynamic range (HDR) illumination from a single RGB image of a reference object. To obtain illumination of a current scene, previous approaches inserted a special camera in that scene, which may interfere with user's immersion, or they analyzed reflected radiances from a passive light probe with a specific type of materials or a known shape. The proposed method does not require any additional gadgets or strong prior cues, and aims to predict illumination from a single image of an observed object with a wide range of homogeneous materials and shapes. To effectively solve this ill-posed inverse rendering problem, three sequential deep neural networks are employed based on a physically-inspired design. These networks perform end-to-end regression to gradually decrease dependency on the material and shape. To cover various conditions, the proposed networks are trained on a large synthetic dataset generated by physically-based rendering. Finally, the reconstructed HDR illumination enables realistic image-based lighting of virtual objects in MR. Experimental results demonstrate the effectiveness of this approach compared against state-of-the-art methods. The paper also suggests some interesting MR applications in indoor and outdoor scenes.

듀얼 카메라가 장착된 이동형 디바이스를 이용한 포커스 콘텍스트 영상 처리 장치 및 그 방법Pat.
우운택 · 김기홍 · 김장윤

KR 10-2074072-0000 · Korea · Granted

본 발명은 듀얼 카메라가 장착된 이동형 디바이스를 이용한 포커스 콘텍스트 영상 처리 장치 및 그 방법에 관한 것으로, 이동형 디바이스의 고화질 카메라로부터 촬영되는 주요 관심 영역의 고화질 영상 데이터, 및 광각 카메라로부터 촬영되는 주변 영역의 광각 영상 데이터를 이용하여 증강현실 영상 콘텐츠를 제공할 수 있다.

손 자세 인식 및 적응적 공간 탐색을 활용한 손 관절 추적 방법 및 그 장치Pat.
우운택 · 박갑용

KR 10-2150794-0000 · Korea · Granted

손 자세 인식 및 적응적 공간 탐색을 활용한 손 관절 추적 방법 및 그 장치가 개시된다. 본 발명의 일 실시예에 따른 손 자세 추적 방법은 손 자세를 포함하는 깊이 이미지를 수신하는 단계; 상기 깊이 이미지에 기초하여 상기 손 자세에 대한 제1 이미지와 제2 이미지를 생성하는 단계; 상기 제1 이미지에 기초하여 상기 제1 이미지에 대응하는 손 대표 자세를 분류하는 단계; 상기 제2 이미지와 상기 분류된 손 대표 자세에 기초하여 미리 설정된 파티클들 각각을 생성하는 단계; 상기 생성된 파티클들 각각에 대한 깊이 오류와 자세 오류를 계산하는 단계; 상기 계산된 깊이 오류와 자세 오류를 기반으로 상기 파티클들 각각과 검색 공간 범위를 업데이트하는 단계; 및 상기 업데이트된 파티클들 각각과 검색 공간 범위에 기초하여 손 자세를 추정하는 단계를 포함한다.

한국HCI학회 학술대회 발표 논문집, pp.1080-1083, 2020

201915

Jaeeun Shin* · Hayun Kim · Callum Parker · Hyung-il Kim · Seoyoung Oh · Woontack Woo†

IEEE International Symposium on Mixed and Augmented Reality (ISMAR) 2019Top-tier

One of the main challenges in creating narrative-driven Augmented Reality (AR) content for Head Mounted Displays (HMDs) is to make them equally accessible and enjoyable in different types of indoor environments. However, little has been studied in regards to whether such content can indeed provide similar, if not the same, levels of experience across different spaces. To gain more understanding towards this issue, we examine the effect of room size and furniture on the player experience of Fragments, a space-adaptive, indoor AR crime-solving game created for the Microsoft HoloLens. The study compares factors of player experience in four types of spatial conditions: (1) Large Room - Fully Furnished; (2) Large Room - Scarcely Furnished; (3) Small Room - Fully Furnished; and (4) Small Room - Scarcely Furnished. Our results show that while large spaces facilitate a higher sense of presence and narrative engagement, fully-furnished rooms raise perceived workload. Based on our findings, we propose design suggestions that can support narrative-driven, space-adaptive indoor HMD-based AR content in delivering optimal experiences for various types of rooms.

Shuo Liu* · Seogsung Jang · Woontack Woo†

IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct) 2019

Augmented reality is a technology which extends measureless contents in the virtual space and makes actual objects not limited in the physical properties. As the most popular cultural media among the campus, the poster have its shortcoming of offering restricted information. In this paper, we introduce a responsive poster-reading system which generates diverse contents step by step with body-based and hand-based interactions, and a design guideline which applies for corresponding posters. The proposed method is able to provide users with a user-friendly, immersive and intuitive experience.

Juyoung Lee* · Shaurye Aggarwal · Jason Wu · Thad Starner · Woontack Woo†

Proceedings of the 23rd International Symposium on Wearable Computers

SelfSync enables rapid, robust initiation of a gesture interface using synchronized movement of different body parts. SelfSync is the gestural equivalent of a hotword such as OK-Google in a speech interface and is enabled by the increasing trend where a user wears two or more wearables, such as a smartwatch, wireless earbuds, or a smartphone. In a user study comparing five potential SelfSync gestures in isolation, our system averages 96%, 98% and 88% for user dependent, user adapted, and user independent accuracy, respectively. For when the user has a phone in a pocket and a smart-watch, we suggest twisting the hand about the wrist while moving the leg with the phone in synchrony left and right. When the user has a head worn device and a smartwatch, we suggest twisting the hand while twisting the head left and right.

Seungwon Kim* · Gun Lee · Weidong Huang · Hayun Kim · Woontack Woo · Mark Billinghurst†

Proceedings of the 2019 CHI Conference on Human Factors in Computing SystemsTop-tier

Many researchers have studied various visual communication cues (e.g. pointer, sketching, and hand gesture) in Mixed Reality remote collaboration systems for real-world tasks. However, the effect of combining them has not been so well explored. We studied the effect of these cues in four combinations: hand only, hand + pointer, hand + sketch, and hand + pointer + sketch, with three problem tasks: Lego, Tangram, and Origami. The study results showed that the participants completed the task significantly faster and felt a significantly higher level of usability when the sketch cue is added to the hand gesture cue, but not with adding the pointer cue. Participants also preferred the combinations including hand and sketch cues over the other combinations. However, using additional cues (pointer or sketch) increased the perceived mental effort and did not improve the feeling of co-presence. We discuss the implications of these results and future research directions.

Hochul Cho* · Jangyoon Kim · Woontack Woo†

IEEE Conference on Virtual Reality and 3D User Interfaces (VR) 2019Top-tier

We present a novel view synthesis method that allows users to experience a large-scale Six-Degree-of-Freedom (6-DOF) virtual environment. Our main contributions are the construction of a large-scale 6-DOF virtual environment using multiple 360 images as well as synthesis of a scene from novel viewpoints. Novel view synthesis from a single 360 image can give free viewpoint experience with full 6-DOF of head motion to players, but the moveable space is limited within a context of the image. We propose a novel view synthesis process that references multiple 360 images via reconstructing a large-scale real world based virtual data map and perform a weighted blending for interpolating multiple novel view images. Our results show that our approach provides a wider area of virtual environment as well as a smooth transition between each reference 360 images.