Related Experiment Video
Updated: Jul 8, 2026
08:32
Examining Online Syntactic Processing of Spoken Complex Sentences in Chinese Using Dual-Modal Interference Tasks
Published on: September 5, 2019
The interplay between the auditory and visual modality for end-of-utterance detection
Pashiera Barkhuysen1, Emiel Krahmer, Marc Swerts
1Communication & Cognition, Faculty of Arts, Tilburg University, P.O. Box 90153, NL-5000 LE Tilburg, The Netherlands.
The Journal of the Acoustical Society of America
|January 8, 2008
Summary
Speakers use both auditory and visual cues to signal the end of their utterances. While audio cues are crucial for quick detection, combining audio-visual information significantly improves accuracy in understanding utterance finality.
Area of Science:
- Psychology
- Linguistics
- Communication Studies
Background:
- Auditory cues like intonation and pausing are known to signal utterance endings.
- The role of visual cues in indicating utterance finality is less understood.
- This study investigates the combined effectiveness of auditory and visual signals in end-of-utterance detection.
Purpose of the Study:
- To determine which modalities (audio, visual, or both) speakers use to signal utterance finality.
- To assess observer sensitivity to these unimodal and bimodal signals.
- To compare the relative contributions of audio-only, vision-only, and audio-visual conditions to end-of-utterance detection.
Main Methods:
- A novel semi-controlled production experiment collected speaker utterances in an interview setting.
- Two perception experiments were conducted: a reaction time experiment (Experiment I) and a classification experiment (Experiment II).
- Stimuli were presented unimodally (audio only, vision only) and bimodally (audio-visual).
Main Results:
- Experiment I showed significantly faster reaction times for end-of-utterance detection with bimodal or audio-only stimuli compared to vision-only.
- No significant difference in reaction time was found between bimodal and audio-only conditions.
- Experiment II revealed significantly better classification performance in the bimodal condition than in either unimodal condition.
Conclusions:
- Both auditory and visual cues contribute to end-of-utterance detection, with audio being more critical for speed.
- Combining audio-visual information enhances accuracy in detecting utterance finality.
- Individual speaker expressivity varies across modalities, suggesting differences in multimodal communication strategies.
More Related Videos
Related Concept Videos
01:17Auditory Perception
The auditory system is essential for sound perception, utilizing various critical structures. When sound waves enter the outer ear, they travel through the ear canal and cause the eardrum to vibrate. These vibrations are then transmitted to the middle ear, where three tiny bones – the malleus, incus, and stapes – amplify the sound. This amplification is crucial, as it ensures that the sound vibrations are strong enough to be conveyed to the inner ear. These vibrations then reach the cochlea, a...
01:31Hearing
When we hear a sound, our nervous system is detecting sound waves—pressure waves of mechanical energy traveling through a medium. The frequency of the wave is perceived as pitch, while the amplitude is perceived as loudness.
01:15Auditory Pathway
Auditory pathways constitute the complex neural circuits responsible for transmitting and interpreting auditory information from the peripheral auditory system to the brain. Sound waves are initially captured by the outer ear, funneled through the ear canal, and reach the tympanic membrane (eardrum). These vibrations are transmitted via the middle ear's ossicles to the inner ear's cochlea.
When viewed cross-sectionally, the cochlea reveals the scala vestibuli and scala tympani flanking the...
When viewed cross-sectionally, the cochlea reveals the scala vestibuli and scala tympani flanking the...
01:21Perceiving Loudness, Pitch, and Location
The human brain perceives pitch through two primary mechanisms reflected in place theory and frequency theory. Each mechanism describes how sound waves are interpreted as specific pitches by the brain, offering insights into the intricate processes of auditory perception.
Place theory, or place coding, suggests that different pitches are heard because various sound waves activate specific locations along the cochlea's basilar membrane. The brain determines the pitch of a sound by identifying...
Place theory, or place coding, suggests that different pitches are heard because various sound waves activate specific locations along the cochlea's basilar membrane. The brain determines the pitch of a sound by identifying...
01:29Non-Verbal Cues
Non-verbal communication extends beyond gestures and facial expressions to include vocal elements known as paralanguage. Paralanguage consists of non-verbal vocal cues such as pitch, loudness, speech rate, pauses, and non-verbal vocalizations like laughter, sighs, and moans. These elements not only accompany speech but also provide critical emotional and contextual information.The Role of Paralanguage in CommunicationParalanguage adds depth to spoken language by conveying emotions and...
01:01Perception of Sound Waves
The human ear is not equally sensitive to all frequencies in the audible range. It may perceive sound waves with the same pressure but different frequencies as having different loudness. Moreover, the perception of sound waves depends on the health of an individual's ears, which decays with age. The health of one's ears may also be affected by regular exposure to loud noises.
The pitch of a sound depends on the frequency and the pressure amplitude of the source. Two sounds of the same frequency...
The pitch of a sound depends on the frequency and the pressure amplitude of the source. Two sounds of the same frequency...