completed

스팀

초시공간 교류를 위한 AR-AI-인지-건축 융합연구

Research on personalized virtual agents and spatial fusion technologies that enable people to interact beyond conventional temporal and spatial boundaries.

Overview

Interdisciplinary research on personalized virtual agents and spatial fusion technologies that enable interaction beyond conventional temporal and spatial boundaries.

Final Research Goal
This research aims to develop core technologies for spatiotemporal interaction by integrating augmented reality (AR), artificial intelligence (AI), cognitive science, and architecture. The project consists of two main directions. The first is interaction across time, which supports communication between people in different time contexts through personalized virtual agents that learn and reproduce previously recorded knowledge and behavior. The second is interaction across space, which enables users in physically separated environments to interact within dynamically fused shared spaces.

Focus Areas

Multimodal scene recognition and scene sequence extraction from human activity data
Scene graph–based structuring of people, objects, actions, and spatiotemporal relationships
Personalized virtual agents for question answering, demonstration, performance assessment, and feedback
Behavior and motion generation reflecting individual characteristics
Spatial fusion across physically separated real environments
Adaptive spatial content generation based on user activity and affective states
Cognitive and social factors underlying human–agent and human–human interaction in augmented environments

In the personalized-agent research, multimodal data acquired from an instructor’s activities—such as speech, video, and motion—are transformed into structured scenes, organized into scene sequences, and represented in the form of scene graphs. These scene graphs serve as structured representations through which a personalized virtual agent can retrieve and reproduce a person’s past experiences.

Application
The developed technologies can be applied to education and training, remote collaboration, telepresence, and shared XR environments. For example, an instructor’s lecture and practical demonstration can be recorded as video, audio, and motion data and structured into a scene sequence. A personalized virtual instructor can then reproduce the original lecture for new learners, answer questions, replay relevant demonstrations, and eventually provide performance assessment and feedback during practice.

For spatial interaction, physically separated environments can be combined into a shared space, together with virtual spatial elements generated by AI when needed. The resulting environment can dynamically change according to users’ purposes, activities, locations, interests, and affective states.

Expected Contribution
Structuring and Reuse of Human Experience: Multimodal human activity records are organized into scenes, scene sequences, and scene graphs that preserve relationships among people, objects, actions, space, and time.

Knowledge Transfer through Personalized Virtual Agents: A person’s previously recorded knowledge and behavior can be reproduced through a virtual agent and adaptively delivered through question answering and interaction with new users.

Spatial Interaction Beyond Physical Distance: Spatial fusion technologies enable users in different physical locations to participate in shared activities within an integrated augmented environment.

Human-Centered XR Interaction: Cognitive states and social interaction factors are incorporated into the design and evaluation of personalized agents and fused spaces to support more natural and effective interaction.

Motivation

Problem

Human interaction is fundamentally constrained by both time and physical space. A person who is not present at the same time cannot directly provide instruction or demonstrate a past activity, while people in different locations cannot easily share the same physical context.

Although video and motion recordings can preserve parts of past experiences, they do not directly provide the structured information required for an intelligent agent to reuse those experiences. Contextual information such as people, objects, actions, spatial relationships, and event order is often distributed across separate data modalities rather than organized as a coherent representation of experience.

Similarly, remote XR systems can visually connect people in different locations, but differences in the physical environments surrounding each user make it difficult to provide a natural and consistent shared spatial context.

Existing limitations

Existing virtual agents are typically based on general knowledge or static information and do not sufficiently reflect the multimodal experiences of a specific person. Recorded video, audio, and motion data are also difficult to reuse directly because they do not explicitly represent relationships among people, objects, actions, space, and time.

Scene graph research has mainly been used to represent objects and spatial relationships for environment understanding, particularly in intelligent agents and autonomous robotics. In contrast, this research extends scene graphs toward a representation for recording human experiences at the event level and reproducing them along a temporal sequence.

Existing remote and mixed-reality systems have commonly connected spaces by directly aligning two environments, overlapping shared free areas, or augmenting selected remote objects or users into another space. Such approaches provide limited support for flexibly reorganizing multiple spaces according to users’ changing purposes and activities.

Our approach

This project approaches spatiotemporal interaction through two complementary directions: interaction across time and interaction across space.

For interaction across time, multimodal human activity data are analyzed and structured into meaningful scenes. Changes in activity are organized into scene sequences, while 3D spatial information, object attributes and semantics, human actions, object relationships, and temporal order and dependencies among events are represented as a dynamic scene graph.

The scene graph serves as the underlying representation through which a personalized virtual agent can retrieve and reproduce past experiences. When a new user interacts with the agent, the interaction can be incorporated into the scene sequence, and relevant events can be retrieved or reordered according to the user’s location, movement, and interaction context. The representation also includes interpolation information for supplementing missing information between events, allowing contextual continuity to be preserved across different spatial and temporal conditions.

Based on these representations, personalized virtual agents can answer users’ questions, reproduce previously recorded demonstrations, evaluate learner performance, and provide feedback. The project further aims to generate agent behaviors appropriate to the surrounding space using recorded scene sequences and language outputs, and ultimately to generate personalized motion that reflects the characteristic behavior patterns and habits of a specific individual.

For interaction across space, the project combines different real environments according to their geometric and functional characteristics and users’ purposes, while integrating AI-generated virtual spatial elements when needed. The spatial configuration can dynamically change according to users’ locations and interests, and spatial content can adapt to users’ activities and affective states.

Together, these technologies aim to extend the boundaries between past and present and across physically separated environments, enabling interaction beyond time and space through personalized virtual agents and adaptive XR spaces.