Towards Multisensory Perception: Modeling and Rendering Sounds of Tool-Surface Interactions

Area of Science:

• Haptic Perception
• Auditory Feedback
• Computational Auditory Scene Analysis

Background:

• Touch-produced sounds offer rich information about surface properties and user interaction.
• Real-time modeling and rendering of these sounds are crucial for immersive experiences.

Purpose of the Study:

• To develop a statistical learning-based approach for modeling and rendering touch-produced sounds in real time.
• To evaluate the impact of virtual sounds on human perception of texture, especially when combined with haptic cues.

Main Methods:

• A data-driven method using recorded audio signals from tool-surface interactions.
• Segmentation of audio signals, labeling with average velocity, and modeling with wavelet trees via a moving window approach.
• Real-time sound synthesis using user velocity to select tree models and employ breadth-first search with inverse wavelet transform.

Main Results:

• Virtual sounds were synthesized in real time based on user velocity and interaction data.
• User studies demonstrated that virtual sounds, alongside haptic cues, significantly improved the perception of texture roughness and hardness.
• Perception of slipperiness was found to be primarily dependent on touch, with less influence from auditory cues.

Conclusions:

• The proposed statistical learning approach effectively models and renders realistic touch-produced sounds.
• Integrating virtual sounds with haptic feedback enhances the perception of certain texture properties, offering a more complete sensory experience.
• Auditory cues play a differential role in texture perception, with touch being dominant for slipperiness.