Light Field Relighting: A Diffusion-Guided 3D Gaussian Splatting Approach

Light Field image processing, empowered by the advancement in computer vision and graphics, offers valuable domain and angular information. Although it holds great promise for applications like virtual reality (VR), augmented reality (AR) and 3D imaging, one significant hurdle remains: achieving realistic relighting in light field images while maintaining high visual fidelity. The compromised output quality basically resulted from the limited baseline of light field images, which typically hinders their ability to generate accurate 3D proxies and perform effective inverse rendering. In this paper, we present a groundbreaking approach that combines the strengths of 3D Gaussian Splatting (3DGS) and pre-trained Stable Diffusion model to achieve state-of-the-art result in light field images relighting. The normal and depth images generated by the Stable Diffusion model provide crucial guidance for our approach, allowing both 3D modeling and inverse rendering to achieve outstanding results with enhanced accuracy and realism. Additionally, the angular information in light field images also provides extra context that can aid in delivering more accurate and detailed depths and normals. To evaluate our framework, we perform quantitative and qualitative experiments on both the synthetic HCI light field benchmark and the Light-Field Intrinsic Dataset (LFID), encompassing synthetic scenes with ground truth and preliminary validation on real-world light field captures. The results demonstrate that our approach achieves competitive performance in novel view synthesis while outperforming baseline methods in intrinsic material decomposition and physical relighting, offering a viable solution for light field inverse rendering.