XR Memory

Much of the knowledge produced in real-world work and learning environments is not captured as complete, structured information. A skilled worker may recognize a subtle change in sound, notice an unusual vibration, shift attention toward a particular component, and make a decision based on years of accumulated experience. Yet conventional recordings capture only fragments of this process—such as video, audio, sensor readings, or system logs—without preserving how these signals relate to one another across space and time.

As a result, what is recorded often represents only what happened, while the surrounding context—where it happened, what objects and people were involved, what preceded the event, and what the user was attending to or trying to accomplish—is easily lost. Reusing these fragmented records for training, knowledge transfer, safety analysis, or immersive reconstruction therefore requires more than simply storing additional data. The data must be organized into a representation that preserves the relationships among people, objects, actions, space, time, and user state.

Current multimodal and XR systems typically process individual data streams independently or combine them only at the level of synchronized playback. Video, audio, motion, physiological signals, and spatial information may be recorded together, but missing segments, sensor noise, tracking failures, and differences in temporal resolution make it difficult to reconstruct a coherent experience.

Digital twins and conventional 3D representations provide spatial models of environments and equipment, but they generally do not capture the semantic and cognitive context surrounding human activity. They represent where objects are and how they are configured, but not necessarily why a worker attended to a particular component, how an action relates to an earlier event, or which part of an experience is important for another user to understand.

Content creation is another barrier. Reconstructing incomplete experiences as interactive XR content typically requires extensive manual editing and specialized development expertise. This limits the ability of workers, trainers, and domain experts to directly preserve and reuse their own knowledge.

XRMemory approaches these challenges by treating an experience as a structured combination of multimodal signals, spatial context, human behavior, and semantic relationships rather than as a collection of independent recordings.

First, heterogeneous data—including video, audio, IMU, location, EEG, and other sensor signals—are temporally and spatially aligned to establish a common representation of an event. AI-based processing is then used to identify relevant objects, actions, people, temporal events, and relationships, organizing them into a Memory Scene Graph (MSG). The MSG provides a structured representation through which fragmented observations can be connected, missing contextual information can be inferred, and relevant memory content can be retrieved according to the user or task.

In parallel, incomplete visual and spatial observations are reconstructed into high-fidelity 3D environments using AI-based spatial reconstruction techniques. Cognitive and behavioral signals are incorporated to represent not only what occurred in the environment but also aspects of how users perceived and responded to it, such as attention, cognitive load, or hazard awareness.

Finally, these representations are connected to an intelligent XR authoring and visualization platform. Natural-language interaction and AI agents allow domain experts and non-expert users to create, edit, and adapt memory-based XR content without conventional programming workflows. The resulting content can then be presented according to the user’s context and needs, enabling past experiences and expert knowledge to be reconstructed as reusable, adaptive XR experiences.