Intrinsic Scene Properties from a Single RGB-D Image

Area of Science:

• Computer Vision
• Computer Graphics
• Computational Imaging

Background:

• Traditional intrinsic image decomposition methods like SIRFS struggle with complex scenes.
• Occlusion and spatially-varying illumination in natural scenes pose significant challenges.
• RGB-D sensors offer depth information that can aid in scene property recovery.

Purpose of the Study:

• To generalize the SIRFS model for improved intrinsic scene property recovery from single RGB-D images.
• To address limitations of existing methods in handling occlusions and complex lighting.
• To leverage depth data from RGB-D sensors for enhanced shape and reflectance estimation.

Main Methods:

• Extended the Shape, Illumination, and Reflectance From Shading (SIRFS) model to Scene-SIRFS.
• Modeled scenes using mixtures of shapes and illuminations within a soft segmentation framework.
• Integrated noisy depth maps from RGB-D sensors to guide shape estimation.

Main Results:

• Developed a technique to recover shape, illumination, reflectance, and shading from a single RGB-D image.
• Generated an improved depth map, surface normals, reflectance, shading, and spatially varying illumination.
• Demonstrated improved performance on natural scenes compared to the original SIRFS model.

Conclusions:

• Scene-SIRFS effectively recovers intrinsic scene properties from single RGB-D images, even in challenging conditions.
• The method provides outputs valuable for computer graphics (relighting) and computer vision (recognition, segmentation).
• Integration of depth data significantly enhances the robustness and accuracy of the recovery process.