Papers & Patents
Publications & Patents
264 outputs
264 outputs
201624
12th International Conference on Signal-Image Technology & Internet-Based Systems (SITIS) 2016
In this paper, we propose an all-in-one mobile outdoor augmented reality (AR) framework for a cultural heritage site. The framework was designed to incorporate computer vision-based augmented reality technology and ontology-based data-authoring technology. Through this framework, we clearly explain how to create 3D visual data for camera pose estimation and how to connect AR content with a cultural heritage site. In addition, we suggest a multi-threading camera tracking and offer an estimation model for mobile AR application. Finally, we have confirmed the efficiency and reliability of our framework. Through this vision-based AR framework, seamless AR application for a cultural heritage site can be made.
12th International Conference on Signal-Image Technology & Internet-Based Systems (SITIS) 2016
In this study, we present a novel Augmented Reality (AR) application for providing contextual information of cultural heritage sites. Whereas most location-based AR systems are designed to show simple information on a Point of Interest (POI) in the real world, the suggested application offers information related to various cultural heritages including tangible and intangible heritages linked to the POI. is is accomplished by a cultural heritage ontology that aggregates heterogeneous data and reorganizes information in themes and relationships. The users can browse information such as the person who created the POI and events that took place at the location as separate entities. We implemented the application for Injeongjeon and the vicinity of Changdeokgung palace and conducted user studies to determine how people explore and consume contextual information at the heritage site. We provide directions that will be useful for designing information to support heritage site visiting, and discuss how the AR technology could improve the application.
Proceedings of the 18th ACM International Conference on Multimodal Interaction
Orbits combines a visual display and an eye motion sensor to allow a user to select between options by tracking a cursor with the eyes as the cursor travels in a circular path around each option. Using an off-the-shelf Jins MEME pair of eyeglasses, we present a pilot study that suggests that the eye movement required for Orbits can be sensed using three electrodes: one in the nose bridge and one in each nose pad. For forced choice binary selection, we achieve a 2.6 bits per second (bps) input rate at 250ms per input. We also inntroduce Head Orbits, where the user fixates the eyes on a target and moves the head in synchrony with the orbiting target. Measuring only the relative movement of the eyes in relation to the head, this method achieves a maximum rate of 2.0 bps at 500ms per input. Finally, we combine the two techniques together with a gyro to create an interface with a maximum input rate of 5.0 bps.
Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct
Solar System implements smooth pursuit eye movement interactions on commercial smart glasses using electrooculography. The system requires no calibration and little to no training. We present a prototype implementation, describe initial user tests and show several application scenarios for hands-free eye gaze interactions.
Proceedings of the 33rd Computer Graphics International
We propose an efficient method for model-based 3D tracking of hand articulations observed from an egocentric viewpoint that aims at supporting the manipulation of virtual objects. Previous model-based approaches optimize non-convex objective functions defined in the 26 Degrees of Freedom (DoFs) space of possible hand articulations. In our work, we decompose this space into six articulation subspaces (6 DoFs for the palm and 4 DoFs for each finger). We also label each finger with a Gaussian model that is propagated between successive image frames. As confirmed by a number of experiments, this divide-and-conquer approach tracks hand articulations more accurately than existing model-based approaches. At the same time, real time performance is achieved without the need of GPGPU processing. Additional experiments show that the proposed approach is preferable for supporting the accurate manipulation of virtual objects in VR/AR scenarios.
Distributed, Ambient and Pervasive Interactions
The Augmented Reality applications have received great attention in the recent years. However, there is still a lack of formal description of such systems currently. In this paper, we propose a new formal model for context-aware semantic Augmented Reality systems. The model can be divided into two parts: a set-theory function based method allows the formalization of an Augmented Reality system while an integrated time-space-motion logic provides the description of the behavior of the system. The suggested model enables the characterization of an Augmented Reality system with mathematical precision. In addition, logical inferences can be performed by means of the logic part of the formal description. The practical applicability of the proposed model is shown through use cases.
Distributed, Ambient and Pervasive Interactions
In this paper, we present a unified framework for remote collaboration using interactive augmented reality (AR) authoring and hand tracking methods. The proposed framework enables a local user to organize AR digital contents for making a shared working environment and collaborate multiple users in the distance. To develop the framework, we combine two core technologies: (i) interactive AR authoring method utilizing a smart input device for making a shared working space, (ii) hand-augmented object interaction method by tracking two hands in egocentric camera view. We implement a prototype of the proposed remote collaboration framework for testing its feasibility in an indoor environment. To the end, we expect that our framework enables collaboration as feeling a sense of co-presence with remote users in a user’s friendly AR working space.
Human Interface and the Management of Information: Applications and Services
In this paper, we introduce the K-Culture Time Machine Project, which develops a mobile AR platform for visualizing time-space connected cultural contents of Korea. Existing AR Applications in cultural heritage domains are currently not interoperable and cannot reuse content. To solve this problem, we developed the modified AR reference model as generic framework of a context-aware AR platform; and we developed the context-aware AR platform with several core technologies according to this model. For back-end, we established the Korean Cultural Heritage Data Model (KCHDM) to aggregate the heterogeneous cultural heritage databases in Korea. We also developed the semi-auto time-space correlation generation module for domain experts. Moreover, we developed the authoring tool to generate the time-space connected AR contents. For the front-end, we developed the vision- and sensor-based spatial data composition technology to perform the solid tracking in outdoor environment and context-aware AR framework. Through the UI/UX and 3D contents, whole technologies are packaged into the mobile AR platform. As a validation process for the application, a mobile AR application for the Korean world cultural heritage was developed. Based on this project, an interoperable AR platform that responds with heterogeneous database would be developed and smart tour guide of the cultural heritage site would be possible.
Distributed, Ambient and Pervasive Interactions
We present novel face landmark detection and tracking methods which are independent of user facial differences in a scenario of Spatial Augmented Reality (SAR) interaction. The proposed methods do not require a preliminary general face model to detect or track landmarks. Our contributions include: (i) fast face landmark detection, which is achieved based on our modified Latent Regression Forest (LRF) and (ii) model-independent facial landmark tracking by revising outliers based on a direction and displacement of neighboring landmarks. We also discuss (iii) feature enhancements based on RGB and depth images for supporting several interaction scenarios in SAR environments. We anticipate that the proposed methods promise several interesting scenarios, even under severe head orientation in SAR interaction without wearing any wearable devices.
Proceedings of the 2016 CHI Conference on Human Factors in Computing SystemsTop-tier
Virtual fitting rooms equipped with magic mirrors let people evaluate fashion items without actually putting them on. The mirrors superimpose virtual clothes on the user's reflection. We contribute the Mirror Mirror system, which not only supports mixing and matching of existing fashion items, but also lets users design new items in front of the mirror and export designs to fabric printers. While much of the related work deals with interactive cloth simulation on live user data, we focus on collaborative design activities and explore various ways of designing on the body with a mirror.
IEEE Second International Conference on Multimedia Big Data (BigMM) 2016
In recent years, augmented-reality (AR) has been attracting extensive attentions from both the research community and industry as a new form of media, mixing virtual content into the physical world. However, the scarcity of the AR content and the lack of user contexts are major impediments to providing and representing rich and dynamic multimedia content on AR applications. In this study, we propose an approach to search and filter big multimedia data, specifically geo-tagged mobile videos, for context-aware AR applications. The challenge is to automatically search for interesting video segments out of a huge amount of user-generated mobile videos, which is one of the biggest multimedia data, to be efficiently incorporated into AR applications. We model the significance of video segments as AR content adopting camera shooting patterns defined in filming, such as panning, zooming, tracking and arching. Then, several efficient algorithms are proposed to search for such patterns using fine granular geospatial properties of the videos such as camera locations and viewing directions over time. Experiments with real-world geo-tagged video dataset show that the proposed algorithms effectively search for a large collection of user-generated mobile videos to identify top K significant video segments.
IEEE Symposium on 3D User Interfaces (3DUI) 2016
We introduce a smartwatch assisted sensor fusion approach to robustly track 6-DOF hand movement in head mounted display (HMD) based augmented reality (AR) environment, which can be used for robust 3D object manipulation. Our method uses a wrist-worn smartwatch with HMD-mounted depth sensor to robustly track 3D position and orientation of user's hand. We introduce HMD-based augmented reality platform with smartwatch, and method to accurately calibrate orientation between smartwatch and HMD. We also implement natural 3D object manipulating system using 6-DOF hand tracker with hand grasping detection. Our proposed system is easy to use, and doesn't require any hand held devices.
Proceedings of the 7th Augmented Human International Conference 2016
This paper proposes a system which enables users to have enhanced interior deigning and authoring tool through augmented reality (AR). The proposed system, which we refer to as DreamHouse, focuses on providing natural user interaction and realistic AR experience by enabling following features: 1) allowing users to utilize bare-hand interaction with an attached egocentric RGB-D camera when trying to make detail adjustments to virtual objects' size, location and orientation, 2) rendering every virtual object in consideration of an environment and lighting conditions, creating photo-realistic scenes to help users to have immersive and realistic interior designing experience. As a result, DreamHouse allows users to freely and easily interact in physical space with virtual objects using bare hands and gives immersive and realistic AR authoring experience to users through photo-realistic rendering.
Asian Workshop on User Interface(ACM AUI 2016), Oct, 2016
Asian Workshop on User Interface(ACM AUI 2016), Oct, 2016
Journal of Virtual Reality Society of Japan, Vol.21, no.2, pp.126-130, Jun, 2016
KR 10-1663470-0000 · Korea · Granted
본 발명은 콘텐츠 서비스 지원을 위한 서비스 수행 시 역사문화 관련 스토리텔링 콘텐츠 생성을 지원하기 위한 문화해설 서비스 모드가 실행되면, 하나 이상의 사료 정보 서비스 서버로부터 데이터 제공을 요청하고 이를 수집하는 과정과, 상기 다수의 사료 정보 서비스 서버로부터 수집된 사료 정보를 기설정된 콘텐츠 서비스 정책 기반 온톨로지 모델링(Ontology Modeling)을 통해 온톨로지 네트워크로 구축하여 DB화하는 과정과, 상기 DB화된 사료 정보를 상기 문화해설 서비스 모드에서 선택된 저작 인터페이스 모드를 통해 스토리텔링 기반 제작자별 적응적 콘텐츠를 생성하여 상기 스토리텔링 콘텐츠 검색 기반 상호 작용형 콘텐츠 서비스를 제공하는 과정을 포함함을 특징으로 한다.
KR 10-1687017-0000 · Korea · Granted
본 발명은 사용자가 착용형 증강 현실 (wearable AR) 환경에서 맨 손 (Bare Hand)으로 가상 3D 객체를 조작할 수 기술에 관한 것으로, 특히 공간(환경)에 설치된 별도의 손, 카메라 추적 장치 없이도, RGB-D 카메라의 거리 입력 데이터를 이용하여, 공간에서 착용형 디스플레이착용형 디스플레이리 카메라 쌍의 3차원 위치와 사용자의 손의 3차원 위치를 알아내고 이를 기반으로 사용자의 맨 손 상호작용이 가능하도록 하는 기술이다.
KR 10-1677426-0000 · Korea · Granted
본 발명은 사용자 인터럽트(interrupt)를 통해 잠금 해제 인터페이스를 디스플레이하는 과정과, 상기 잠금 해제 인터페이스를 통해 입력된 잠금 해제 패턴을 수신하는 과정과, 상기 수신된 잠금 해제 패턴을 검출하고 검출된 잠금 해제 패턴에 대응하는 모드를 실행하여 상기 검출된 잠금 해제 패턴에 대응하는 오브젝트의 상태 정도를 측정하는 과정과, 기설정되어 분류된 오브젝트 유형에 따라 유형별 적정 상태 정도 범위가 측정되어 매칭된 룩업 테이블을 호출하여 상기 측정된 오브젝트의 상태 정도를 판단하고, 판단 결과를 상기 잠금 해제 인터페이스를 통해 피드백하는 과정을 포함함을 특징으로 한다.
KR 10-1666561-0000 · Korea · Granted
본 발명은 디스플레이 장치에서 표시되는 3D 이미지에서 사용자의 동작에 의한 이벤트가 발생되면, 상기 사용자의 동작에 대응하는 제1 입체형상의 제1 부분 공간을 획득하는 타겟팅(Targeting)부 및 상기 타겟팅부가 획득한 제1 부분 공간의 범위 내에서 사용자의 제스쳐에 따라 위치와 스케일이 조절되는 제2 입체형상의 제2 부분 공간을 획득하는 개선부를 포함한다.
KR 10-1687309-0000 · Korea · Granted
본 발명은 HMD(Head Mounted Display)가 구비된 단말의 정보 제공 서비스 방법에 있어서, 제1 사용자의 HMD 뷰잉(viewing) 영역에서 사용자 인터럽트를 통해 감지된 이벤트에 대응하여 기설정된 메뉴를 표시하는 과정과, 표시된 상기 메뉴로부터 적어도 하나 이상의 타 사용자간 화면 공유 서비스를 지원하기 위한 항목을 활성화하여 페어링(pairing)된 타 사용자의 HMD에 디스플레이 하고자 하는 컨텐츠를 브로드캐스팅하는 과정과, 상기 제1 사용자의 HMD로부터 브로드캐스팅되어 적어도 하나 이상의 HMD에서 공유된 콘텐츠가 디스플레이된 화면의 뷰잉 영역에서 상기 HMD 사용자별 상호작용을 통해 제어되는 과정을 포함함을 특징으로 한다.
IEEE Transactions on Human-Machine Systems
We present a novel natural user interface framework, called Meta-Gesture, for selecting and manipulating rotatable virtual reality (VR) objects in egocentric viewpoint. Meta-Gesture uses the gestures of holding and manipulating the tools of daily use. Specifically, the holding gesture is used to summon a virtual object into the palm, and the manipulating gesture to trigger the function of the summoned virtual tool. Our contributions are broadly threefold: 1) Meta-Gesture is the first to perform bare hand-gesture-based orientation-aware selection and manipulation of very small (nail-sized) VR objects, which has become possible by combining a stable 3-D palm pose estimator (publicly available) with the proposed static-dynamic (SD) gesture estimator; 2) the proposed novel SD random forest, as an SD gesture estimator can classify a 3-D static gesture and its action status hierarchically, in a single classifier; and 3) our novel voxel coding scheme, called layered shape pattern, which is configured by calculating the fill rate of point clouds (raw source of data) in each voxel on the top of the palm pose estimation, allows for dispensing with the need for preceding hand skeletal tracking or joint classification while defining a gesture. Experimental results show that the proposed method can deliver promising performance, even under frequent occlusions, during orientation-aware selection and manipulation of objects in VR space by wearing head-mounted display with an attached egocentric-depth camera (see the supplementary video available at: http://ieeexplore.ieee.org).
IEEE Transactions on Human-Machine Systems
In this paper, we propose TunnelSlice, which enables natural acquisition of subspace in an augmented scene from an egocentric view, even for scenarios involving ambiguous center objects or object occlusion. In wearable augmented reality (AR), approaching a three-dimensional (3-D) region including the objects of interest has become more important than approaching distant objects one by one. However, existing ray-based volumetric selection through a head worn display accompanies difficulties in defining a desired 3-D region due to obstacles by occlusion and depth perception. The proposed TunnelSlice effectively determines a cuboid transform, excluding unnecessary areas of a user-defined tunnel via two-handed pinch-based procedural slicing from an egocentric view. Through six scenarios involving central object status and different occlusion levels, we conducted a user study of TunnelSlice. Compared with two existing approaches, TunnelSlice was preferred by the subjects and showed greater stability for all scenarios, and outperformed the other approaches in a scenario involving strong occlusion without a central object. TunnelSlice is thus expected to serve as a key technology for spatial protocol and interaction using a subspace in wearable AR.
KR 10-1678510-0000 · Korea · Granted
증강 현실 기반의 의상 디자인 장치는 판상의 반투명한 거울층 및 상기 거울층 배면에서 영상을 표시하는 디스플레이 계층을 포함하는 표시부, 상기 거울 전면에 위치한 사용자에 대한 깊이 영상을 획득하는 깊이 센서부, 상기 거울을 향하는 상기 사용자의 일면 중 적어도 일부의 타겟 영역에 의상의 디자인을 구성하는 적어도 하나의 디자인 요소를 투사하는 프로젝터부 및 상기 깊이 센서부가 획득한 영상을 이용하여 상기 사용자의 위치를 추적하고, 상기 프로젝터부가 상기 타겟 영역에 상기 디자인 요소를 투사하도록 제어하는 제어부를 포함한다.
