GANG: Geometrically-Aligned Neural Gaussians for Efficient and Realistic Relighting

Efficient and realistic relighting of complex scenes with unknown illumination remains a crucial but challenging task. Recent advancements in 3D Gaussian Splatting (3DGS) have shown impressive object-level relighting. However, they still struggle with complex real-world scenes, mainly due to the challenges of accurately decoupling intricate geometry, materials, and lighting using concise 3D Gaussian primitives. In this paper, we propose a new Geometrically-Aligned Neural Gaussian Splatting (GANG) method, which performs efficient physically based rendering (PBR) directly on anchor-based relightable neural Gaussians. Our key idea is to regularize the decoded neural Gaussians geometrically aligned with the latent signed distance field (SDF) surface spawned from anchors using a differentiable implicit indicator function (IIF) solver. It brings effective geometric association to accurate decoupling of materials and lighting for efficient and realistic relighting of complex scenes. Furthermore, we propose a locally consistent geometry regularization to guide more concise neural Gaussian learning with a hybrid lighting model, which combines position-learnable spherical Gaussians (SGs) and an environment map, allowing accurate modeling of both local and global illumination. Experimental results on public datasets demonstrate that GANG consistently outperforms previous PBR methods in material decomposition and relighting quality, while representing complex scenes with concise anchors. To the best of our knowledge, GANG is a new state-of-the-art 3DGS method for realistic relighting, enabling efficient rendering and flexible editing materials and illumination, especially for complex scenes.