Related Experiment Video
Updated: Jul 3, 2026
13:51
Cross-Modal Multivariate Pattern Analysis
Published on: November 9, 2011
Matching auditory and visual signals: is sensory modality just another feature?
Jeroen S Benjamins1, Maarten J van der Smagt, Frans A J Verstraten
1Experimental Psychology Division, Helmholtz Institute, Utrecht University, Heidelberglaan 2, NL-3584 CS Utrecht, The Netherlands.
Perception
|August 9, 2008
Summary
Perceiving the world requires sensory integration. Crossmodal integration has lower temporal limits than unimodal integration, similar to adding features within one sense.
Area of Science:
- Cognitive Neuroscience
- Psychology
- Sensory Perception
Background:
- Coherent perception relies on integrating sensory information.
- Understanding the temporal limits of this integration is crucial.
Purpose of the Study:
- To investigate the temporal limits of sensory integration.
- To compare unimodal (visual, auditory) and crossmodal (auditory-visual) integration.
- To explore the effect of adding sensory modalities on temporal integration limits.
Main Methods:
- Participants performed unimodal and crossmodal stimulus matching tasks.
- Alternating visual and auditory stimuli were presented at various rates (up to 4 Hz and higher).
- Temporal stimulus characteristics like offsets and stimulus onset asynchrony (SOA) were manipulated.
Main Results:
- Crossmodal auditory-visual matching failed at approximately 4 Hz, while unimodal matching succeeded at higher rates.
- Performance was not affected by manipulating stimulus offsets or SOAs.
- The temporal limit difference between crossmodal and unimodal integration mirrored that of integrating additional features within a single modality.
Conclusions:
- Integrating information across different sensory modalities (crossmodal) imposes stricter temporal limitations than within a single modality (unimodal).
- Adding a sensory modality for integration has a comparable cognitive load to integrating an additional feature within a single modality.
- These findings offer insights into the neural mechanisms of sensory integration and perception.
Related Concept Videos
01:15Sensory Modalities
Sensation typically is the process by which the sensory receptors and sense organs detect stimuli from the internal and external environment and transmit this information to the central nervous system for processing.
General senses refer to the broad category of sensory information detected by receptors in the body and can be further grouped into somatic and visceral senses. Somatic sensations include touch, pressure, temperature, and pain and are essential for navigating our environment and...
General senses refer to the broad category of sensory information detected by receptors in the body and can be further grouped into somatic and visceral senses. Somatic sensations include touch, pressure, temperature, and pain and are essential for navigating our environment and...
01:26Introduction to Special Senses
Sensory receptors play an integral part in comprehending our external and internal environments. They receive diverse stimuli, converting them into the nervous system's electrochemical signals. This conversion occurs as the stimulus alters the sensory neuron's cell membrane potential, instigating the generation of an action potential. This action potential is subsequently transmitted to the central nervous system (CNS), which integrates with other sensory data or higher cognitive functions.
01:31What is a Sensory System?
Sensory systems detect stimuli—such as light and sound waves—and transduce them into neural signals that can be interpreted by the nervous system. In addition to external stimuli detected by the senses, some sensory systems detect internal stimuli—such as the proprioceptors in muscles and tendons that send feedback about limb position.
01:26Visual System
Light enters the eye through the cornea, a transparent, dome-shaped surface covering the surface of the eyeball that helps to direct and focus incoming light. This light is then channeled toward the pupil, an adjustable opening whose size is controlled by the iris. The iris, a pigmented muscle, regulates the amount of light entering the eye by contracting or dilating the pupil, thereby ensuring optimal light levels for clear vision.
Once through the pupil, the light passes through the lens, a...
Once through the pupil, the light passes through the lens, a...
01:17Auditory Perception
The auditory system is essential for sound perception, utilizing various critical structures. When sound waves enter the outer ear, they travel through the ear canal and cause the eardrum to vibrate. These vibrations are then transmitted to the middle ear, where three tiny bones – the malleus, incus, and stapes – amplify the sound. This amplification is crucial, as it ensures that the sound vibrations are strong enough to be conveyed to the inner ear. These vibrations then reach the cochlea, a...
01:21Perceiving Loudness, Pitch, and Location
The human brain perceives pitch through two primary mechanisms reflected in place theory and frequency theory. Each mechanism describes how sound waves are interpreted as specific pitches by the brain, offering insights into the intricate processes of auditory perception.
Place theory, or place coding, suggests that different pitches are heard because various sound waves activate specific locations along the cochlea's basilar membrane. The brain determines the pitch of a sound by identifying...
Place theory, or place coding, suggests that different pitches are heard because various sound waves activate specific locations along the cochlea's basilar membrane. The brain determines the pitch of a sound by identifying...