Event-Based Video Reconstruction With Deep Spatial-Frequency Unfolding Network.

Area of Science:

• Computer Vision
• Signal Processing
• Deep Learning

Background:

• Current event-based video reconstruction methods struggle with brightness and structural decoupling, causing distortion.
• Existing methods often require computationally expensive models like Transformers for non-local information acquisition.

Purpose of the Study:

• To propose a novel network, Deep Spatial-Frequency Unfolding Reconstruction Network (DSFURNet), for event-based video reconstruction.
• To address limitations in spatial-domain methods by incorporating frequency domain analysis.

Main Methods:

• Developed a variational model with three regularization terms: brightness (Fourier amplitudes), structure (Fourier phases), and initialization (event to frame conversion).
• Designed spatial-frequency domain approximation operators for efficient integration of local and global information.
• Unfolded the optimization algorithm into an iterative deep network (DSFURNet) for continuous constraint application.

Main Results:

• DSFURNet effectively integrates local spatial and global frequency information at a lower computational cost.
• The iterative network design allows continuous application of regularization constraints, progressively improving reconstructed video quality.
• Achieved significant reductions in network parameters while enhancing evaluation metrics compared to existing methods.

Conclusions:

• DSFURNet offers an efficient and effective solution for event-based video reconstruction by utilizing the frequency domain.
• The proposed method overcomes key challenges like exposure distortion and computational expense associated with prior techniques.
• This approach demonstrates the potential of spatial-frequency analysis in advancing event-based vision.