Related Experiment Video
Updated: Nov 28, 2025
04:32
Sound Source Localization Testing in Single-sided Deafness Following Bone Conduction Intervention
Published on: December 20, 2024
655
DMMAN: A two-stage audio-visual fusion framework for sound separation and event localization
Ruihan Hu1, Songbing Zhou1, Zhi Ri Tang2
1Institute of Intelligent Manufacturing, Guangdong Academy of Sciences, Guangdong Key Laboratory of Modern Control Technology, Guangzhou, China.
Summary
This study introduces the Deep Multi-Modal Attention Network (DMMAN) to separate and localize sounds in videos. DMMAN effectively handles mixed audio, improving sound source separation and event localization accuracy.
Area of Science:
- Computer Science
- Signal Processing
- Artificial Intelligence
Background:
- Videos contain mixed sounds, making it difficult to distinguish individual sound sources.
- Current methods struggle with separating and localizing sounds in unconstrained video datasets.
Purpose of the Study:
- To develop a model for sound source separation and event localization in videos.
- To address the challenge of mixed audio in multimedia content.
Main Methods:
- Introduced the Deep Multi-Modal Attention Network (DMMAN) for unconstrained video datasets.
- Utilized a multi-modal separator and matching classifier with two-stage fusion of audio-visual features.
- Employed regression and classification losses for model training.
Main Results:
- DMMAN achieved high-quality sound source separation, validated by Signal-to-Distortion Ratio and Signal-to-Interference Ratio metrics.
- The model demonstrated effectiveness in mixed sound scenes previously unaddressed.
- Achieved superior classification accuracy for event localization compared to baseline methods.
Conclusions:
- DMMAN successfully separates sound sources and localizes events in videos.
- The model offers a robust solution for complex audio-visual analysis tasks.
- This approach advances the capabilities of multimedia content understanding.
Related Concept Videos
01:21Perceiving Loudness, Pitch, and Location
668
The human brain perceives pitch through two primary mechanisms reflected in place theory and frequency theory. Each mechanism describes how sound waves are interpreted as specific pitches by the brain, offering insights into the intricate processes of auditory perception.
Place theory, or place coding, suggests that different pitches are heard because various sound waves activate specific locations along the cochlea's basilar membrane. The brain determines the pitch of a sound by...
Place theory, or place coding, suggests that different pitches are heard because various sound waves activate specific locations along the cochlea's basilar membrane. The brain determines the pitch of a sound by...
668
01:01Perception of Sound Waves
5.2K
The human ear is not equally sensitive to all frequencies in the audible range. It may perceive sound waves with the same pressure but different frequencies as having different loudness. Moreover, the perception of sound waves depends on the health of an individual's ears, which decays with age. The health of one's ears may also be affected by regular exposure to loud noises.
The pitch of a sound depends on the frequency and the pressure amplitude of the source. Two sounds of the same...
The pitch of a sound depends on the frequency and the pressure amplitude of the source. Two sounds of the same...
5.2K
01:15Auditory Pathway
6.6K
Auditory pathways constitute the complex neural circuits responsible for transmitting and interpreting auditory information from the peripheral auditory system to the brain. Sound waves are initially captured by the outer ear, funneled through the ear canal, and reach the tympanic membrane (eardrum). These vibrations are transmitted via the middle ear's ossicles to the inner ear's cochlea.
When viewed cross-sectionally, the cochlea reveals the scala vestibuli and scala tympani flanking...
When viewed cross-sectionally, the cochlea reveals the scala vestibuli and scala tympani flanking...
6.6K
01:17Auditory Perception
813
The auditory system is essential for sound perception, utilizing various critical structures. When sound waves enter the outer ear, they travel through the ear canal and cause the eardrum to vibrate. These vibrations are then transmitted to the middle ear, where three tiny bones – the malleus, incus, and stapes – amplify the sound. This amplification is crucial, as it ensures that the sound vibrations are strong enough to be conveyed to the inner ear. These vibrations then reach the...
813
00:53Sound Waves: Interference
4.2K
Sound waves can be modeled either as longitudinal waves, wherein the molecules of the medium oscillate around an equilibrium position, or as pressure waves. When two identical waves from the same source superimpose on each other, the combination of two crests or two troughs results in amplitude reinforcement known as constructive interference. If two identical waves, that are initially in phase, become out of phase because of different path lengths, the combination of crests with troughs...
4.2K
01:30Classification of Signals
1.2K
In signal processing, signals are classified based on various characteristics: continuous-time versus discrete-time, periodic versus aperiodic, analog versus digital, and causal versus noncausal. Each category highlights distinct properties crucial for understanding and manipulating signals.
A continuous-time signal holds a value at every instant in time, representing information seamlessly. In contrast, a discrete-time signal holds values only at specific moments, often denoted as x(n), where...
A continuous-time signal holds a value at every instant in time, representing information seamlessly. In contrast, a discrete-time signal holds values only at specific moments, often denoted as x(n), where...
1.2K