DiFaReli++: Diffusion Face Relighting With Consistent Cast Shadows.

Area of Science:

• Computer Vision
• Computer Graphics
• Artificial Intelligence

Background:

• Single-view face relighting is challenging due to global illumination and cast shadows.
• Existing methods often rely on error-prone intrinsic decomposition (3D shape, albedo, lighting).
• These methods require extensive training data, including ground-truth lighting and light stages.

Purpose of the Study:

• To develop a novel approach for single-view face relighting in the wild.
• To bypass the need for accurate intrinsic estimation and extensive ground-truth data.
• To achieve realistic relighting with temporally consistent cast shadows.

Main Methods:

• Leveraging a conditional diffusion implicit model (DDIM) for decoding disentangled light encoding.
• Utilizing off-the-shelf estimators for 3D shape and facial identity.
• Proposing a novel conditioning technique using rendered shading reference and shadow maps to modulate the DDIM.
• Implementing a single-shot relighting framework.

Main Results:

• The method bypasses the need for intrinsic estimation and can be trained solely on 2D images.
• Achieved state-of-the-art performance on the Multi-PIE benchmark.
• Outperformed the teacher model across all metrics in a single-shot framework.
• Demonstrated realistic relighting with temporally consistent cast shadows.

Conclusions:

• The proposed method offers a robust and data-efficient solution for single-view face relighting.
• The novel conditioning technique effectively models light-geometry interactions.
• The single-shot framework achieves superior performance and realism in challenging in-the-wild scenarios.