Jinwoo Park

In mixed reality (MR), augmenting virtual objects consistently with real-world illumination is one of the key factors that provide a realistic and immersive user experience. For this purpose, we propose a novel deep learning-based method to estimate high dynamic range (HDR) illumination from a single RGB image of a reference object. To obtain illumination of a current scene, previous approaches inserted a special camera in that scene, which may interfere with user's immersion, or they analyzed reflected radiances from a passive light probe with a specific type of materials or a known shape. The proposed method does not require any additional gadgets or strong prior cues, and aims to predict illumination from a single image of an observed object with a wide range of homogeneous materials and shapes. To effectively solve this ill-posed inverse rendering problem, three sequential deep neural networks are employed based on a physically-inspired design. These networks perform end-to-end regression to gradually decrease dependency on the material and shape. To cover various conditions, the proposed networks are trained on a large synthetic dataset generated by physically-based rendering. Finally, the reconstructed HDR illumination enables realistic image-based lighting of virtual objects in MR. Experimental results demonstrate the effectiveness of this approach compared against state-of-the-art methods. The paper also suggests some interesting MR applications in indoor and outdoor scenes.

This paper proposes a system which enables users to have enhanced interior deigning and authoring tool through augmented reality (AR). The proposed system, which we refer to as DreamHouse, focuses on providing natural user interaction and realistic AR experience by enabling following features: 1) allowing users to utilize bare-hand interaction with an attached egocentric RGB-D camera when trying to make detail adjustments to virtual objects' size, location and orientation, 2) rendering every virtual object in consideration of an environment and lighting conditions, creating photo-realistic scenes to help users to have immersive and realistic interior designing experience. As a result, DreamHouse allows users to freely and easily interact in physical space with virtual objects using bare hands and gives immersive and realistic AR authoring experience to users through photo-realistic rendering.

This paper presents an augmented reality (AR) authoring system that enables an ordinary user to easily build an AR environment by manipulating and placing 3D virtual objects. The system tracks users’ hand motions via an RGB-D camera which built-in an optical see-through (OST) head mounted display (HMD), and interactive features applied to virtual objects by the pre-defined hand gestures. In addition, the virtual objects and dynamic paths are placed by using a smartphone in a real-world space. To implement this system, three cored technologies are needed: (i) segmentation of regions of spaces and real-world objects (ii) hand tracking and gesture recognition for manipulating virtual objects, (iii) dynamic path placing of virtual objects using a smartphone wearing an OST HMD. We implement a prototype of the proposed system for testing its feasibility. To the end, we expect that our system enables simplify AR technology usage to ordinary users who are unfamiliar with professional AR authoring tools.

본 발명은 물리 기반의 딥러닝 네트워크를 통해 재질 정보를 제거하여 대상 객체에서 반사되는 순수한 빛 정보로부터 보다 정확한 조명 조건을 추정하는 기술에 관한 것으로, 입력 이미지에 포함된 단일 재질의 대상 객체를 검출하며, 상기 대상 객체로부터 재질 정보를 제거하여 빛 반사 정보를 획득하는 단계, 상기 대상 객체에서 반사되는 상기 빛 반사 정보에서 파노라마 이미지 형태의 LDR(Low Dynamic Range) 빛 정보를 추정하는 단계 및 상기 LDR 빛 정보를 이용하여 HDR(High Dynamic Range) 빛 정보를 추정하며, 상기 LDR 빛 정보 및 상기 HDR 빛 정보를 합하여 상기 대상 객체에 대한 전방향 빛 정보를 추정하는 단계를 포함한다.

본 발명은 의미론적 객체 매칭을 통한 파노라마 텍스처 매핑 방법 및 시스템에 관한 것으로, 거리 뷰 이미지에서 의미론적 정보를 추출하는 단계, 사용자의 움직임에 따른 카메라 뷰 이미지에 대한 G-버퍼를 렌더링하며, 상기 G-버퍼에서 각 픽셀을 상기 거리 뷰 이미지로 매핑하는 파노라마 이미지 매핑을 수행하는 단계 및 상기 의미론적 정보와 상기 G-버퍼를 이용하여 깊이 테스트, 의미 객체 매칭 테스트 및 3D 인페인팅(inpainting)을 통해 상기 카메라 뷰 이미지와 상기 거리 뷰 이미지에 대한 파노라마 텍스처 매핑을 수행하는 단계를 포함한다.