Haptic-Sound Analysis of Materials' Clustering Based on Tool Tip Tapping Exploration

Area of Science:

• Multimodal sensory perception
• Robotics and artificial intelligence
• Materials science and engineering

Background:

• Tapping surfaces generates both auditory (sound) and tactile (haptic) feedback.
• Understanding how these signals are perceived and integrated is crucial for tasks like object recognition.

Purpose of the Study:

• To investigate how sound and acceleration signals individually contribute to surface texture assessment.
• To determine the effect of integrating both sound and haptic modalities on texture assessment.
• To explore the relationship between sound and haptic signal features across different object types.

Main Methods:

• Texture assessment framed as an unsupervised learning problem.
• Design of perceptual filter banks based on Weber's law for both sound and haptic modalities.
• Development of a symmetric KL divergence-based metric to compare sound and haptic signal similarities.
• Utilizing permutation tests to analyze the complementarity of sound and haptic information.

Main Results:

• Proximity between object types is preserved across both sound and haptic modalities.
• The similarity metric indicates a consistent representation of object properties across sensory inputs.
• Complementarity of sound information to haptic information is object-dependent, as shown by permutation tests.

Conclusions:

• Both sound and haptic signals provide valuable information for individual surface assessment.
• Integration of sound and haptic information enhances texture assessment capabilities.
• The study highlights the varying degrees of complementarity between auditory and tactile information for different object types.