Sound-to-Touch Crossmodal Pitch Matching for Short Sounds

Area of Science:

• Crossmodal perception research
• Auditory and tactile sensory science
• Human-computer interaction

Background:

• Audio-to-tactile conversion enhances user experience in various applications.
• Understanding crossmodal congruence is key for effective sensory substitution.

Purpose of the Study:

• To establish a functional relationship between sound loudness spectra and tactile vibration frequencies.
• To develop a generalizable sound-to-touch crossmodal pitch mapping function.
• To explore spectral matching rules between auditory and tactile stimuli.

Main Methods:

• Synthesizing single-frequency amplitude-modulated vibrations to match sound intensity and temporal characteristics.
• Evaluating crossmodal congruence for 175 sound-vibration pairs across varying frequencies.
• Estimating a mapping function from sound loudness spectra to harmonious vibration frequencies using a dataset.

Main Results:

• A functional relationship was estimated, mapping sound loudness spectra to optimal tactile pitch (vibration frequency).
• Cross-validation confirmed the reliability of the developed sound-to-touch crossmodal pitch mapping function.
• This research provides the first general rules for spectral matching between sound and touch.

Conclusions:

• The study successfully created a method for mapping sound characteristics to tactile sensations.
• The findings contribute to designing more intuitive and congruent multi-sensory user experiences.
• This work pioneers the exploration of generalizable spectral correspondence between auditory and tactile domains.