Minju Baeck

We propose a visuo-tactile feedback method that combines virtual hand visualization and fingertip vibrations to modulate affective roughness perception in VR. While prior work has focused on object-based textures and vibrotactile feedback, the role of visual feedback on virtual hands remains underexplored. Our approach introduces affective visual cues including line shape, motion, and color applied to hand outlines, and examines their influence on both affective responses (arousal, valence) and perceived roughness. Results show that sharp contours enhanced perceived roughness, increased arousal, and reduced valence, intensifying the emotional impact of haptic feedback. In contrast, color affected valence only, with red consistently lowering emotional positivity. These effects were especially noticeable at lower haptic intensities, where visual cues extended affective modulation into mid-level perceptual ranges. Overall, the findings highlight how integrating expressive visual cues with tactile feedback can enrich affective rendering and offer flexible emotional tuning in immersive VR interactions.

We propose a collaborative scene mood authoring system in a virtual reality (VR) environment, where multiple users design and control the multimodal feedback (i.e., audiovisual and haptic) of a shared scene using voice commands. To support this, the user’s command is processed by large language models (LLMs) with the scene metadata and mood presets to identify intended target objects and mood, and the system modifies the objects to match the desired atmospheric expression. To cover a wide range of affective moods in scene mood authoring, we devised four distinct mood presets that represent different combinations of arousal and valence across the affective space. Each preset is expressed through a set of labeled visual, auditory, and haptic feedback elements, enabling coherent and multimodal expression of affective states within the scene.

We propose a system that provides personalized interpretation of physical paintings using an AR cursor in a mixed reality (MR) environment. In galleries, pointing at specific regions of complex paintings is often restricted, making it hard to ask targeted questions. While prior works have proposed interactive systems to assist visitor understanding, they typically offered generalized explanations that were not grounded in real, physically exhibited artworks. To address this gap, our approach allows users to intuitively drag across the surface of a painting using a hand-guided AR cursor to specify a Region of Interest. The cropped image, along with the user’s spoken question, is then sent to a Vision Language Model (VLM), which generates a personalized and context-aware explanation based on the selected area. This approach enables users to gain a more precise and meaningful understanding of the painting, guided by their specific interests.

In collaborative VR, asymmetric access to haptic hardware creates a critical information gap: tactile evidence remains private to the haptic user, hindering the shared understanding needed for joint decision-making. While prior work has explored crossmodal sensory cues in virtual environments, it remains unclear how such cues should be designed for asymmetric collaboration, where collaborators receive information through different modalities. In our setting, the haptic user feels roughness through fingertip vibration, whereas the non-haptic user relies on vision alone. To reduce this asymmetry, we propose externalizing an object’s tactile state through a glanceable hand-outline visual proxy. Specifically, we examine whether abstract visual roughness cues based on line shape and motion can encode three discrete roughness levels for both haptic and non-haptic users. Two preliminary studies establish a shared visual semantics by identifying visually distinguishable cues for non-haptic users and validating their visuo-haptic correspondence for haptic users. In a main study of a collaborative sorting task, showing this visualization on both users’ hands significantly reduced completion time relative to a no-visualization baseline. Moreover, NU-side cue visibility was associated with higher confidence and perceived contribution for the non-haptic user. These findings show that hand-anchored abstract visual cues provide a lightweight means of externalizing object tactile state, reducing information asymmetry without compromising social presence.

Immersive Computational Notebooks (ICoN) extend traditional notebook environments into immersive spaces, enabling analysts to interact with multimodal artifacts, including code, narratives, data tables, and visualizations. By integrating multimodal artifacts into a single immersive workspace, ICoN enables analysts to transition between analytical tasks seamlessly. Meanwhile, understanding organizational strategies is critical for designing effective interactions to further support analysts. However, prior research on immersive computational notebooks has primarily examined organizational strategies centered on single-modality artifacts. Systematic investigations of how analysts spatially organize the complex relationships among multimodal artifacts in a single immersive workspace remain underexplored. To address this gap, we conducted a user study to examine organizational strategies for multimodal artifacts in immersive computational notebooks. Our findings show that participants predominantly adopted depth-based layouts, and their spatial organization was largely structured around cell-based artifacts.

We propose GestureMark, a novel input technique for target selection on XR glasses using smartwatch touch gestures as input. As XR glasses get smaller and lighter, their usage increases rapidly, leading to a higher demand for efficient shortcuts for everyday life. We explored the uses of gesture input on smartwatch touchscreen, including simple swipe, swipe combinations, and bezel-to-bezel (B2B) gesture as an input modality. Through an experiment with 16 participants, we found that while swipe gestures were efficient for four-choice selections, B2B was superior for 16-choice inputs. Feedback mechanisms did not enhance performance but reduced perceived workload. Our findings highlight the potential of integrating smartwatches as secondary input devices for XR glasses.