SfSNet: Learning Shape, Reflectance and Illuminance of Faces in the Wild

Area of Science:

• Computer Vision
• Computer Graphics
• Machine Learning

Background:

• Accurate decomposition of human face images into intrinsic properties like shape, reflectance, and illuminance is crucial for various computer vision and graphics applications.
• Existing methods often struggle with unconstrained images and capturing high-frequency details.

Purpose of the Study:

• To introduce SfSNet, an end-to-end deep learning framework for accurate face decomposition.
• To develop SfSMesh, a companion network for 3D face mesh reconstruction using SfSNet's outputs.
• To improve upon state-of-the-art methods in inverse rendering and 3D face reconstruction.

Main Methods:

• SfSNet employs a novel decomposition architecture with residual blocks, learning from both synthetic and real-world images.
• It utilizes a photometric reconstruction loss to capture low-frequency variations and high-frequency details.
• SfSNet separates albedo and normal, then predicts lighting using these and the original image.

Main Results:

• SfSNet achieves significantly better quantitative and qualitative results in inverse rendering and intrinsic decomposition compared to existing methods.
• The companion network, SfSMesh, reconstructs 3D face meshes with superior accuracy on real-world images.
• The framework effectively handles unconstrained human face images.

Conclusions:

• SfSNet provides an accurate and robust framework for decomposing human face images into shape, reflectance, and illuminance.
• The integrated SfSMesh network demonstrates improved 3D face mesh reconstruction capabilities.
• This work advances the state-of-the-art in intrinsic image decomposition and 3D face modeling.