Reconstructing Reflection Maps Using a Stacked-CNN for Mixed Reality Rendering.

Area of Science:

• Computer Vision
• Computer Graphics
• Human-Computer Interaction

Background:

• Realistic integration of virtual objects in augmented and mixed reality (AR/MR) requires matching lighting and reflectance between real and virtual elements.
• Existing methods struggle to accurately capture and represent real-world environmental lighting for seamless AR/MR experiences.

Purpose of the Study:

• To develop a method for reconstructing real-world environmental lighting, encoded as a reflection map (RM), from conventional photographs.
• To enable high-fidelity rendering of virtual objects that realistically match the lighting conditions of real-world scenes in AR/MR applications.

Main Methods:

• A stacked convolutional neural network (SCNN) was proposed to predict high dynamic range (HDR) 360° reflection maps (RMs) with varying roughness from low dynamic range photographs.
• The SCNN was progressively trained from high to low roughness to generate RMs suitable for rendering virtual objects with different material properties (diffuse to glossy).

Main Results:

• The method successfully reconstructed environmental lighting and generated RMs that allowed for high-fidelity rendering of virtual objects.
• Virtual objects rendered using the predicted RMs showed plausible integration and composition within both indoor and outdoor scenes.
• A comparative user study and error metrics indicated improved quality over previous state-of-the-art methods.

Conclusions:

• The proposed SCNN-based method effectively reconstructs real-world lighting from single photographs for AR/MR applications.
• This approach significantly enhances the realism and visual coherence of virtual objects integrated into real-world environments.
• The method offers a promising solution for improving spatial presence and immersion in AR and MR experiences.