Rain Removal From Light Field Images With 4D Convolution and Multi-Scale Gaussian Process.

Area of Science:

• Computer Vision
• Image Processing
• Machine Learning

Background:

• Single-image deraining is challenging due to limited information.
• Light Field Images (LFIs) offer rich 3D scene data but are underutilized for deraining.
• Existing methods struggle to fully leverage LFI's multi-view and depth information.

Purpose of the Study:

• To propose a novel method, 4D-MGP-SRRNet, for effective rain streak removal from LFIs.
• To exploit the abundant 3D information embedded in LFIs for improved deraining.
• To address the limitations of single-image deraining methods.

Main Methods:

• Utilizing 4D convolutional layers to process all sub-views of an LFI simultaneously.
• Introducing MGPDNet with a Multi-scale Self-guided Gaussian Process (MSGP) module for high-resolution rain streak detection.
• Employing semi-supervised learning for MSGP training on both synthetic and real-world rainy LFIs.
• Integrating a 4D convolution-based Depth Estimation Residual Network (DERNet) for depth map estimation and fog map generation.
• Leveraging an adversarial recurrent neural network for progressive rain streak elimination and LFI restoration.

Main Results:

• Demonstrated superior performance in rain streak removal from LFIs compared to existing methods.
• Successfully utilized multi-view and depth information from LFIs for enhanced deraining.
• Achieved high-quality restoration of rain-free LFIs on both synthetic and real-world datasets.

Conclusions:

• The proposed 4D-MGP-SRRNet method effectively removes rain streaks from LFIs by leveraging their rich 3D information.
• The integration of 4D convolutions, advanced rain detection, and depth estimation significantly improves deraining performance.
• This work offers a promising direction for robust image restoration in adverse weather conditions using LFIs.