Generating Visually Aligned Sound from Videos.

Area of Science:

• Computer Vision
• Audio Signal Processing
• Machine Learning

Background:

• Generating realistic audio synchronized with video content is a complex challenge.
• Existing models struggle with sounds originating outside the camera's view, leading to misaligned audio-visual mappings.
• This can result in generated sounds that are temporally or contextually inconsistent with the visual input.

Purpose of the Study:

• To develop a robust framework for generating temporally and content-wise aligned sound from natural videos.
• To address the challenge of irrelevant sounds from outside the video frame interfering with sound generation.
• To improve the accuracy and realism of synthesized audio in response to visual stimuli.

Main Methods:

• Proposed a framework named REGNET for video-to-sound generation.
• Extracted appearance and motion features from video frames to identify sound-emitting objects.
• Introduced an innovative audio forwarding regularizer using real sound as input for stronger training supervision.
• Removed the regularizer during testing for purely visual feature-based sound production.

Main Results:

• REGNET significantly enhances both temporal and content-wise alignment between generated sound and video.
• Extensive evaluations demonstrated the effectiveness of the proposed audio forwarding regularizer.
• The generated sound achieved a remarkable 68.12% success rate in fooling human evaluators.

Conclusions:

• REGNET effectively tackles the challenge of generating accurate and aligned sound from video.
• The audio forwarding regularizer successfully prevents the model from learning incorrect visual-sound mappings.
• The framework demonstrates a significant advancement in audio-visual synthesis, with potential applications in multimedia and virtual reality.