Learning Frame-Event Fusion for Motion Deblurring.

Area of Science:

• Computer Vision
• Signal Processing
• Machine Learning

Background:

• Motion deblurring is an ill-posed problem due to information loss during blurring.
• Event cameras offer high-speed, asynchronous motion data that can complement traditional image sensors.
• Integrating information from multiple sensor types is crucial for advanced deblurring.

Purpose of the Study:

• To propose a novel frame-event fusion framework for event-driven motion deblurring.
• To effectively fuse complementary information from traditional frames and event camera data.
• To address modality redundancy and capture long-range spatio-temporal dependencies for improved deblurring.

Main Methods:

• A frame-event fusion framework (FEF-Deblur) is developed, modeling cross-modal fusion as complementary-unique feature separation and aggregation.
• Parallel intra-modal self-attention and inter-modal cross-attention are used to infer unique and complementary features.
• A correlation-based constraint is introduced for redundancy suppression, and recurrent cross-attention preserves spatio-temporal dependencies.

Main Results:

• The FEF-Deblur framework effectively separates and aggregates complementary and unique features, suppressing modality redundancy.
• Recurrent cross-attention successfully preserves spatio-temporal information, crucial for motion deblurring.
• Extensive experiments show superior performance over state-of-the-art event-based and image/video-based deblurring methods on synthetic and real-world datasets.

Conclusions:

• The proposed FEF-Deblur method demonstrates a significant advancement in event-driven motion deblurring by effectively fusing frame and event data.
• The framework's ability to handle modality redundancy and capture long-range spatio-temporal interactions leads to superior deblurring performance.
• The approach offers a promising direction for enhancing motion deblurring using multi-modal sensor data.