Collaborative Scene Mood Authoring with Voice-driven Multimodal Feedback Design in Virtual Reality

Abstract

We propose a collaborative scene mood authoring system in a virtual reality (VR) environment, where multiple users design and control the multimodal feedback (i.e., audiovisual and haptic) of a shared scene using voice commands. To support this, the user’s command is processed by large language models (LLMs) with the scene metadata and mood presets to identify intended target objects and mood, and the system modifies the objects to match the desired atmospheric expression. To cover a wide range of affective moods in scene mood authoring, we devised four distinct mood presets that represent different combinations of arousal and valence across the affective space. Each preset is expressed through a set of labeled visual, auditory, and haptic feedback elements, enabling coherent and multimodal expression of affective states within the scene.