Learning visual to auditory sensory substitution reveals flexibility in image to sound mapping
Asa Kucinkas1,2, Chrysa Retsa1,2, Peter B L Meijer3
1The Radiology Department, Lausanne University Hospital and University of Lausanne, Lausanne, Switzerland.
NPJ Science of Learning
|December 3, 2025
Summary
Structured mappings enhance learning for visual-to-auditory sensory substitution devices (SSDs). Both traditional and reversed spatial mappings improved performance over arbitrary controls, showing flexibility in SSD design.
Area of Science:
- Neuroscience
- Human-Computer Interaction
- Sensory Science
Background:
- Visual-to-auditory sensory substitution devices (SSDs) convert visual information into sound.
- The vOICe system maps visual pixels to sound features like pitch and time.
- Existing mappings often prioritize technical preservation of visual data over intuitive crossmodal correspondences.
Purpose of the Study:
- To investigate the role of crossmodal correspondences in learning SSDs.
- To compare the efficacy of traditional (pitch-elevation, time-azimuth) versus reversed (pitch-azimuth, time-elevation) spatial mappings.
- To evaluate learning with a structured mapping versus an arbitrary control mapping.
Main Methods:
- Sixty sighted adults were randomly assigned to Traditional, Reversed, or Control groups.
- Participants learned visual-to-auditory mappings using simplified stimuli.
- Learning and recognition of novel stimuli were assessed after a 30-minute training period.
Main Results:
- Both Traditional and Reversed mapping groups learned significantly faster than the Control group.
- No significant difference in learning or performance was found between the Traditional and Reversed groups.
- Structured spatial mappings, regardless of specific configuration, facilitate SSD learning.
Conclusions:
- The ability to learn SSDs is enhanced by structured spatial mappings.
- The mapping of visual spatial information onto acoustic features appears flexible, not strictly bound by innate crossmodal correspondences.
- SSDs can be customized based on user needs and specific applications.
Related Concept Videos
01:01Neuroplasticity
1.5K
Neuroplasticity reflects the brain's remarkable capacity to adapt and evolve, responding dynamically to learning, experiences, or injury by reorganizing its neural circuitry. This reorganization involves creating new neural connections and refining old ones through a series of biological processes that contribute to the brain's lifelong development and adaptability.
1.5K
01:15Auditory Pathway
7.0K
Auditory pathways constitute the complex neural circuits responsible for transmitting and interpreting auditory information from the peripheral auditory system to the brain. Sound waves are initially captured by the outer ear, funneled through the ear canal, and reach the tympanic membrane (eardrum). These vibrations are transmitted via the middle ear's ossicles to the inner ear's cochlea.
When viewed cross-sectionally, the cochlea reveals the scala vestibuli and scala tympani flanking...
When viewed cross-sectionally, the cochlea reveals the scala vestibuli and scala tympani flanking...
7.0K
01:17Auditory Perception
998
The auditory system is essential for sound perception, utilizing various critical structures. When sound waves enter the outer ear, they travel through the ear canal and cause the eardrum to vibrate. These vibrations are then transmitted to the middle ear, where three tiny bones – the malleus, incus, and stapes – amplify the sound. This amplification is crucial, as it ensures that the sound vibrations are strong enough to be conveyed to the inner ear. These vibrations then reach the...
998
01:01Perception of Sound Waves
5.4K
The human ear is not equally sensitive to all frequencies in the audible range. It may perceive sound waves with the same pressure but different frequencies as having different loudness. Moreover, the perception of sound waves depends on the health of an individual's ears, which decays with age. The health of one's ears may also be affected by regular exposure to loud noises.
The pitch of a sound depends on the frequency and the pressure amplitude of the source. Two sounds of the same...
The pitch of a sound depends on the frequency and the pressure amplitude of the source. Two sounds of the same...
5.4K
01:15Sensory Modalities
3.7K
Sensation typically is the process by which the sensory receptors and sense organs detect stimuli from the internal and external environment and transmit this information to the central nervous system for processing.
General senses refer to the broad category of sensory information detected by receptors in the body and can be further grouped into somatic and visceral senses. Somatic sensations include touch, pressure, temperature, and pain and are essential for navigating our environment and...
General senses refer to the broad category of sensory information detected by receptors in the body and can be further grouped into somatic and visceral senses. Somatic sensations include touch, pressure, temperature, and pain and are essential for navigating our environment and...
3.7K
01:21Perceiving Loudness, Pitch, and Location
896
The human brain perceives pitch through two primary mechanisms reflected in place theory and frequency theory. Each mechanism describes how sound waves are interpreted as specific pitches by the brain, offering insights into the intricate processes of auditory perception.
Place theory, or place coding, suggests that different pitches are heard because various sound waves activate specific locations along the cochlea's basilar membrane. The brain determines the pitch of a sound by...
Place theory, or place coding, suggests that different pitches are heard because various sound waves activate specific locations along the cochlea's basilar membrane. The brain determines the pitch of a sound by...
896