Physically-inspired Deep Light Estimation from a Homogeneous-Material Object for Mixed Reality Lighting.

Area of Science:

• Computer Vision
• Computer Graphics
• Artificial Intelligence

Background:

• Realistic mixed reality (MR) requires virtual objects to match real-world lighting.
• Existing methods for estimating scene illumination often require specialized equipment or strong prior assumptions.
• These limitations hinder immersion and practical application in MR environments.

Purpose of the Study:

• To develop a novel deep learning-based method for estimating high dynamic range (HDR) illumination from a single RGB image.
• To enable realistic image-based lighting of virtual objects in MR without intrusive hardware or complex priors.
• To address the ill-posed inverse rendering problem for illumination estimation.

Main Methods:

• A novel deep learning approach using three sequential neural networks is proposed.
• The networks are designed with a physically-inspired framework to progressively reduce material and shape dependency.
• Training was conducted on a large synthetic dataset generated via physically-based rendering.

Main Results:

• The proposed method effectively estimates HDR illumination from a single object image.
• Experimental results show superior performance compared to state-of-the-art illumination estimation techniques.
• The reconstructed HDR illumination facilitates realistic virtual object lighting in MR.

Conclusions:

• The developed deep learning method offers a practical solution for accurate HDR illumination estimation in MR.
• This approach enhances user immersion by seamlessly integrating virtual objects with real-world lighting.
• Potential applications span both indoor and outdoor MR scenarios.