A Unified Framework for Event-Based Frame Interpolation With Ad-Hoc Deblurring in the Wild

Area of Science:

• Computer Vision
• Machine Learning
• Robotics

Background:

• Effective video frame interpolation relies on accurate motion handling.
• Existing methods often assume sharp input frames, neglecting motion-induced blur.
• Event-based vision offers asynchronous data that can complement traditional frames.

Purpose of the Study:

• To develop a unified framework for event-based frame interpolation that addresses both sharp and blurry video inputs.
• To improve the generalization of event-based interpolation models to real-world data.
• To introduce a challenging high-resolution dataset for evaluating event-based interpolation and deblurring.

Main Methods:

• A bidirectional recurrent network fuses input frames and event data adaptively.
• The framework incorporates an integrated deblurring capability.
• Self-supervised learning is employed to enhance domain transfer from synthetic to real data.
• A new high-resolution dataset, HighREV, is introduced.

Main Results:

• The proposed method outperforms state-of-the-art approaches in frame interpolation, single image deblurring, and joint tasks.
• Self-supervised training significantly reduces performance gaps between synthetic and real-world datasets.
• The HighREV dataset provides a robust benchmark for challenging event-based vision tasks.

Conclusions:

• The unified framework effectively handles motion blur in event-based frame interpolation.
• Self-supervised learning is crucial for real-world applicability of event-based vision models.
• The HighREV dataset facilitates future research in high-fidelity event-based video processing.