Related Experiment Video
Updated: Sep 12, 2025
12:39
A Methodology for Capturing Joint Visual Attention Using Mobile Eye-Trackers
Published on: January 18, 2020
7.8K
UniAV: Unified Audio-Visual Perception for Multi-Task Video Event Localization
Summary
UniAV unifies temporal action localization, sound event detection, and audio-visual event localization for holistic video understanding. This novel framework outperforms specialized models and naive multi-task approaches across benchmarks.
Area of Science:
- Computer Vision
- Machine Learning
- Artificial Intelligence
Background:
- Video event localization encompasses temporal action localization (TAL), sound event detection (SED), and audio-visual event localization (AVEL).
- Current methods often overspecialize in individual tasks, hindering a comprehensive understanding of video content.
- Existing task-specific datasets exhibit significant disparities in size, domain, and duration, complicating unified approaches.
Purpose of the Study:
- To develop a unified framework for simultaneously addressing TAL, SED, and AVEL tasks.
- To facilitate holistic video understanding by integrating knowledge across different event types and modalities.
- To overcome the challenges posed by distinct task characteristics and dataset disparities in existing methods.
Main Methods:
- Introduction of UniAV, a Unified Audio-Visual perception network.
- Development of a unified audio-visual encoder for generic representations across multiple temporal scales.
- Design of task-specific experts to capture unique knowledge for each task.
- Implementation of a novel unified language-aware classifier with semantic-aligned task prompts for flexible, open-set localization.
Main Results:
- UniAV significantly outperforms single-task models and naive multi-task baselines across all three localization tasks.
- The unified architecture effectively learns and shares knowledge across tasks and modalities.
- Superior or on-par performance is achieved compared to state-of-the-art task-specific methods on ActivityNet 1.3, DESED, and UnAV-100.
- The model demonstrates impressive open-set localization capabilities for novel categories.
Conclusions:
- UniAV offers an effective unified framework for multi-task video event localization.
- The proposed architecture enhances holistic video understanding by integrating diverse event information.
- UniAV represents a significant advancement in audio-visual perception and event localization research.
Related Concept Videos
01:21Perceiving Loudness, Pitch, and Location
426
The human brain perceives pitch through two primary mechanisms reflected in place theory and frequency theory. Each mechanism describes how sound waves are interpreted as specific pitches by the brain, offering insights into the intricate processes of auditory perception.
Place theory, or place coding, suggests that different pitches are heard because various sound waves activate specific locations along the cochlea's basilar membrane. The brain determines the pitch of a sound by...
Place theory, or place coding, suggests that different pitches are heard because various sound waves activate specific locations along the cochlea's basilar membrane. The brain determines the pitch of a sound by...
426
01:20Parallel Processing
227
The brain processes sensory information rapidly due to parallel processing, which involves sending data across multiple neural pathways at the same time. This method allows the brain to manage various sensory qualities, such as shapes, colors, movements, and locations, all concurrently. For instance, when observing a forest landscape, the brain simultaneously processes the movement of leaves, the shapes of trees, the depth between them, and the various shades of green. This enables a quick and...
227