RealLiFe: Real-Time Light Field Reconstruction via Hierarchical Sparse Gradient Descent.

Area of Science:

• Computer Vision
• Computer Graphics
• Extended Reality (XR)

Background:

• Real-time light field reconstruction is crucial for Extended Reality (XR) applications.
• Current offline methods offer quality at the cost of speed, while online methods lack generalizability or performance.
• The sparse manifold of Multi-plane Images (MPI) presents an opportunity for accelerated reconstruction.

Purpose of the Study:

• To introduce RealLiFe, a novel method for real-time light field reconstruction from sparse inputs.
• To leverage the sparse nature of MPI for efficient and high-quality light field generation.
• To significantly improve upon the speed and performance of existing light field reconstruction techniques.

Main Methods:

• Introduced RealLiFe, a light field optimization method utilizing Hierarchical Sparse Gradient Descent (HSGD).
• Employed a 3D Convolutional Neural Network (CNN) to generate an initial coarse MPI.
• Optimized the coarse MPI using sparse MPI gradients for rapid refinement.

Main Results:

• RealLiFe achieves comparable visual quality to state-of-the-art offline methods.
• The proposed method is on average 100x faster than existing offline techniques.
• RealLiFe demonstrates superior performance over other online approaches, with approximately 2 dB higher PSNR.

Conclusions:

• RealLiFe enables high-quality, real-time light field reconstruction from sparse inputs.
• The Hierarchical Sparse Gradient Descent (HSGD) approach effectively utilizes MPI sparsity for acceleration.
• This method offers a significant advancement for real-time applications in Extended Reality (XR).