Hierarchical Bayesian Inverse Lighting of Portraits with a Virtual Light Stage

From a single RGB image of an unknown face, taken under unknown conditions, we estimate a physically plausible lighting model. First, the 3D geometry and texture of the face are estimated by fitting a 3D Morphable Model to the 2D input. With this estimated 3D model and a Virtual Light Stage (VLS), we generate a gallery of images of the face with all the same conditions, but different lighting. We consider non-lambertian reflectance and non-convex geometry to handle more realistic illumination effects in complex lighting conditions. Our hierarchical Bayesian approach automatically suppresses inconsistencies between the model and the input. It estimates the RGB values for the light sources of a VLS to reconstruct the input face with the estimated 3D face model. We discuss the relevance of the hierarchical approach to this minimally constrained inverse rendering problem and show how the hyperparameters can be controlled to improve the results of the algorithm for complex effects, such as cast shadows. Our algorithm is a contribution to single image face modeling and analysis, provides information about the imaging condition and facilitates realistic reconstruction of the input image, relighting, lighting transfer and lighting design.