FAV-DenoiseNet: An Audio-Visual Speech Enhancement Framework Based on Conditional Flow Matching and Visual Encoding

Area of Science:

• Speech processing
• Artificial intelligence
• Signal processing

Background:

• Diffusion-based methods offer high performance in audio-visual speech enhancement but suffer from high latency.
• Real-time deployment of speech enhancement systems is hindered by computational costs.
• Existing methods struggle to balance enhancement quality with inference speed.

Purpose of the Study:

• To develop an efficient audio-visual speech enhancement framework (FAV-DenoiseNet) that overcomes the latency limitations of diffusion models.
• To improve the quality of enhanced speech by effectively integrating visual cues.
• To achieve real-time performance without compromising restoration accuracy.

Main Methods:

• A two-stage framework combining discriminative prior denoising and conditional residual flow matching.
• The first stage suppresses noise and provides a stable speech prior.
• The second stage estimates the residual using single-step flow matching with multi-scale cross-modal attention and a residual-controlled fusion strategy.

Main Results:

• FAV-DenoiseNet achieves state-of-the-art performance on benchmark datasets (VoxCeleb2, GRID) with high PESQ, ESTOI, and SI-SDR scores.
• The proposed method demonstrates a low real-time factor (RTF) of 0.086, enabling efficient inference.
• The framework effectively balances speech enhancement quality, detail restoration, and real-time processing.

Conclusions:

• FAV-DenoiseNet successfully addresses the latency and computational cost issues of diffusion-based speech enhancement.
• The proposed two-stage approach with residual compensation and cross-modal attention provides superior audio-visual speech enhancement.
• The framework offers a promising solution for real-time audio-visual speech enhancement applications.