Light field image super-resolution based on raw data with transformers.

Area of Science:

• Computer Vision
• Image Processing
• Artificial Intelligence

Background:

• Light field (LF) image super-resolution (SR) aims to enhance spatial resolution by leveraging complementary information from multiple perspectives.
• Existing LF-SR methods often overlook information loss during raw-to-RGB conversion and underutilize explicit structural information.
• There is a need for methods that process raw LF data and explicitly integrate multiview information for improved SR.

Purpose of the Study:

• To develop a novel LF image super-resolution method that addresses limitations of current approaches.
• To introduce a data generation pipeline for LF raw data suitable for SR tasks.
• To propose an end-to-end deep learning architecture for effective LF image SR.

Main Methods:

• A data generation pipeline was created to collect and process LF raw data for SR.
• An end-to-end convolutional neural network, LF-RawSR, was designed for LF image SR.
• The architecture incorporates an aggregated module with a volume transformer and plane sweep volume for angular information fusion, followed by a cross-view transformer for nonlocal dependency utilization.

Main Results:

• The proposed LF-RawSR method demonstrates superior performance compared to state-of-the-art LF-SR techniques.
• The method successfully restores fine details and clear structures in the super-resolved LF images.
• The approach achieves competitive results with comparable computational costs.

Conclusions:

• The developed LF-RawSR method effectively enhances LF image resolution by utilizing raw data and explicit multiview information.
• The proposed architecture provides a robust framework for LF image SR, outperforming existing methods.
• This work contributes to advancing LF image SR by addressing information loss and improving multiview feature fusion.